Compiling and running jobs on sepeli

This document provides a short guide on how to compile and run programs on Sepeli. For more in-depth information, please see the the Sepeli user's guide.

Compiling

The Pathscale compiler suite is preferred compiler and should be used for compilation. For compatibility purposes Sepeli also contains the PGI and GNU compiler suites.

Overview of compilers

The following chart lists some key characteristics of each compiler suite.

	PathScale	PGI	GNU
Version
Path	/opt/pathscale/	/opt/pgi/	/
C compiler	pathcc	pgcc	gcc
C++ compiler	pathCC	pgCC	g++
F77 compiler	pathf90	pgf77	g77
F90/F95 compiler	pathf90	pgf90	N/A
Optimization flags	-O3 -OPT:Ofast	-O3 -fastsse	-O3 -funroll-all-loops -msse
OpenMP flag	-mp	-mp	N/A
Auto-parallelization flag	N/A	-Mconcur	N/A
Debugger	pathdb	pgdbg	gdb
Link to documentation	link	link	link

Compiling serial programs

64-bit Pathscale Fortran 90 compiler example:

sepeli ~> pathf90 -O3 -OPT:Ofast -o serial.exe serial.f

32-bit code can be compiled with appending _32 to the name of the compiler:

sepeli ~> pathf90_32 -O3 -OPT:Ofast -o serial.exe serial.f

Compiling parallel MPI programs

The recommended primary MPI environment on sepeli is OpenMPI with PathScale compilers in 64-bit mode (openmpi-path64). Binaries compiled with OpenMPI support both InfiniBand and Gigabit Ethernet networks. For more information on OpenMPI, please see the OpenMPI website.

To initialize the MPI environment run:

sepeli ~> use openmpi-path64

After intializing the environment MPI wrapper compilers (mpicc, mpif90, mpiCC) can be used to compile and link the programs. These add the relevant compiler and linker to create MPI binaries.

Further information can be found in the OpenMPI FAQ

Other MPI libraries on sepeli

There are several other MPI libraries on sepeli for compatibility and legacy purposes. These should only be used if there are severe compilation problems with the primary environment or other constraints.

Running

Overview

The batch queuing system in Sepeli is Sun Grid Engine 6.0 update 8 which is the non-commercial version of Sun N1 Grid Engine which is in use on CSC's
Corona application server and the two are very similiar. Note that the parallel environments and queues are different between the tw

Job limits

Parallel processes

By default users can run parallel jobs up to 16 processes.

Memory

Jobs can reserve up to 8GB per process. Sepeli is optimized for jobs which use a maximum of 2GB of resident memory per process. Larger jobs will most
likely have a considerably longer queuing time.

Wall clock time

The current maximum wall clock time soft limit for other than long jobs is 239 hours (SIGUSR1 will be sent) and hard limit is 240 hours (KILL will be
sent). A good estimate of the runtime (option '-l h_rt') of your job will most likely give a shorter queuing time. A default value of 'h_rt' parameter is
8 hours

Serial interactive jobs

IMPORTANT: All files needed by a job must be copied to $WRKDIR, for example the program and input/output files.

Interactive jobs should be run using the qrsh command from the command line. This reserves a single CPU using the batch queuing system and opens a remote
shell to a compute node

#Example: 1 h running time and 1.5 GB memory for an interactive session
#The current maximum wall clock time limit (option -l h_rt)  for interactive jobs is 8 hours.

sepeli /mnt/nas3/wrk/user> qrsh -l h_rt=01:00:00 -now n -l h_vmem=1500M
compute-n-nn ~> myprogram
...
compute-n-nn ~> exit
sepeli ~>

Parallel interactive jobs

IMPORTANT: All files needed by a job must be copied to $WRKDIR, for example the program and input/output files.

Currently only parallel interactive jobs that can run inside a single node are supported

#Example: Running a simple openmpi job.

sepeli /mnt/nas3/wrk/user> qrsh -l h_rt=00:30:00 -now n -pe openmpi-gnu64 4
compute-n-nn ~> use openmpi-gnu64
compute-n-nn ~> mpirun -np 4 myprogram
...
compute-n-nn ~> exit

Serial batch jobs

IMPORTANT: All files needed by a batch job must be copied to $WRKDIR, for example the program and input/output files.

Here is a simple job script for a serial job (myjob.sh). 

#!/bin/sh
# Example job-script-file
#
# Set the name of the job.
#$ -N sge-date-run
# Set the name of the stdout/stderr file
#$ -o serial.out
# Set the initial directory to be the current working directory
#$ -cwd
#Merge the standard out and standard error to one file
#$ -j y
#Inherit the current environment variables
#$ -V
# The max walltime for this job is 31 minutes
#$ -l h_rt=00:31:00
#Memory limit (1.3GB)
#$ -l h_vmem=1300M

./serial.exe

Command qsub [job-script-file] submits a batch job:

sepeli /mnt/nas3/wrk/user> qsub myjob.s

Parallel batch jobs

IMPORTANT. All files needed by a batch job must be copied to $WRKDIR, for example the program and input/output files

This example script works with jobs which use less than 2GB of memory per process. For large memory jobs see the additional instructions below.

#!/bin/sh
# OpenMPI example
# openmpi and 64-bit Pathscale compiler

# Export all environment variables
#$ -V

# Your job name
#$ -N my-openmpi-gnu64-program

# Use current working directory
#$ -cwd

# Join stdout and stder
#$ -j y

# PARALLEL ENVIRONMENT:
# pe request for 64-bit GNU compiled MPICH program. Set your number of processors here
#$ -pe openmpi-gnu64 8

# Enable resource reservation
#$ -R 

# The max hard walltime for this job is 22 minutes (after this it will be killed)
#$ -l h_rt=00:22:00

# The max soft walltime for this job is 21 minute (after this SIGUSR2 will be sent)
#$ -l s_rt=00:21:00

# The following is for reporting only. It is not really needed
# to run the job. It will show up in your output file.
echo "Got $NSLOTS processors.

# The mpirun command.
mpirun -np $NSLOTS my-parallel-openmpi-gnu64-program

Command qsub [job-script-file] submits a batch job:

sepeli /mnt/nas3/wrk/user> qsub myjob.s

Large memory parallel batch jobs (2-8GB per process)

IMPORTANT: The resource parameter h_vmem is deprecated and should not be used.

There are special parallel environments for large memory parallel jobs. To run a large memory job simply add the appropriate postfix to the regular parallel environment:

Memory requirement	PE suffix	Example PE
2GB-2.66GB	*-3G	openmpi-path64-3G
2.66GB-4GB	*-4G	openmpi-path64-4G
4GB-8GB	*-8G	openmpi-path64-8G

Long Lasting jobs

If a serial or parallel batch job takes more than 240 hours there is a queue made for those purposes. Add the following line to the batch script

#$ -l long=true

The maximum wall clock time for those jobs is 504 hours (21 days). The long jobs can not have more than 8 processors.

Useful commands

* man submit qsub, qrsh man pages
    * qsub -help show all qsub options
    * qstat show the status of Grid Engine jobs and queues
    * qstat -f show the full status of Grid Engine jobs and queues
    * qhost -q show the status of Grid Engine hosts
    * qstat -u $USER show the status of my jobs
    * qdel -u $USER delete all my jobs
    * qconf -sc show the list of resources (like h_rt, h_vmem)
    * qconf -spl show all parallel environments
    * qacct -o $USER -j what the scheduling system bookkeeping knows about my ended jobs.
    * qacct -o $USER -j [jobnumber] what the scheduling system bookkeeping knows about a job (qsub command has given the jobnumber