IMPORTANT: All files needed by a job must be copied to $WRKDIR, for example the program and input/output files. Remember to give module load modulefilename commands if needed. Include these commands also in a batch job script if needed.

See section Submitting jobs: bsub for information about the command bsub which is used for launching the batch jobs. Also please note the policies for available amount of cores for each user, in Usage policy and obtaining a user id

This section contains very simple examples of different types. Few examples of how to run parallel batch interactive and real batch jobs are also given in section Running programs (batch and interactive). These show how to use more diverse selection of options.

Options and arguments (including the job command) to bsub can be given on the command line, in a batch job script or by both ways. All following examples can be run interactively (if they are not very long) using the option -I  on the command line or in a batch job script by adding the line:

#BSUB -I

See examples in Chapters Running programs (batch and interactive) and Submitting jobs: bsub.

In interactive case the standard output and error are displayed on the user's terminal if no output files for them are given. In real batch jobs the standard  output and error together with the job report are sent via email to the user (with or without the option -N, because -N is the default in this case), unless the option -o and and/or  -e (and similar) are not given. If the option -N is given with the option  -o then only the job report is sent via email and the standard output (and error) is printed to a file.

NB! Remember to use -M ...  option with bsub to specify the needed amount of memory for your job to prevent swapping and performance regressions. The option -M is mandatory. A batch job start script (esub) sets automatically the LSF-SLURM external schedule option -ext "SLURM[constraint=constraint]", where constraint is smallmem, mediummem, bigmem or hugemem or their combination, according to the setting of -M, that is acccording to requested memory. There is no need to use this external schedule option anymore by yourself.

Parallel HP-MPI jobs

The following example shows how to submit an parallel HP-MPI job and its batch job script (exjob1.csh). The script may be as follows:

#!/bin/csh
#BSUB -n 64
#BSUB -W 2:30
#BSUB -o mpi_prog.out.%J
#BSUB -M 1048576
#BSUB -N
#BSUB -u user1@univ2.fi
 
mpirun -srun ./mpi_prog

This request 64 cores for running the same amount of MPI tasks, one per core (-n 64). Because the default is to provide full nodes with 4 cores, this is equivalent of requesting 16 nodes (=64/4). The requested wall time for the whole job is 2 hours 30 minutes (-W 2:30). The maximum memory requested per process is 1 GB (-M 1048576, 1048576/1024*1024 = 1 GB). So four MPI tasks fit just and just to one small memory node (1 GB/core), if each of them uses 100-200 MB less than 1 GB of memory. This is because operating system takes about 500-600 MB of memory of a compute node. When the job is finished LSF sends an e-mail (-N) to the address user1@univ2.fi (-u user@univ2.fi). You should replace this with your own e-mail address.

This is submitted as follows. This shows also that the bsub command shows the job ID, which is needed later below in some other commands.

user1@c553:/wrk/user1> bsub < exjob1.csh
Job <11636> is submitted to default queue <parallel>.

Please note that the explicit standard input redirection (<) must be used. Otherwise the #BSUB option lines of the script are not interpreted.

If more memory per MPI task is needed and you want  one MPI task to fit in a small memory node (you may still get bigger memory nodes), your batch job script might be:

#!/bin/csh
#BSUB -n 64
#BSUB -W 2:30
#BSUB -o mpi_prog.out.%J
#BSUB -M 4100000 
#BSUB -ext "SLURM[nodes=64]"
mpirun -srun ./mpi_prog

This example illustrates how to submit a parallel job using the LSF-SLURM external scheduler specifying additional SLURM options, in this case requesting 64 nodes (see the section  Submitting jobs: bsub under the heading "LSF-SLURM external schedule options" for explanation of this).

Here slightly less than 4 GB of memory (4100000/1024*1024 = 3.91 GB) is requested per process (per MPI task) and as many nodes as cores, that is 64 nodes and 64 cores. This means that each of the 64 MPI tasks is run in its own node with one core, other three cores being idle in each node.

Parallel serial job (non-MPI parallel job)

In a non-MPI parallel job the same serial program is run simultaneously or "parallel" in all reserved cores (therefore the name "parallel serial job"). To make sense for this kind of job, every single program instance should use its own unique input files, that is, the same program should be run multiple times using different input in each time. This may be accomplished by some programming tricks in a program source code itself and also in the batch job script. This may be based, e.g., on the list of selected host (compute node) names for the job, which are specified in the environment variables LSB_HOSTS and LSB_MCPU_HOSTS, and the processor and core numbers of each host. This may be difficult, and better method for such jobs is an array job, which may be considered to be a special case of a non-MPI parallel job. See Chapter Serial batch jobs. However, for the sake of completeness, the examples for parallel non-MPI jobs are also presented here.

The following example shows how to submit a non-MPI parallel job:

bsub -n 4 -M 1048576 -o program.out srun program

The following example illustrates also how to submit a parallel (serial) job using the LSF-SLURM external scheduler specifying additional SLURM options, in this case requesting four nodes.

bsub -n 4 -M 1048576 -o prog.o%J -ext "SLURM[nodes=4]" srun prog

Here %J is the unique batch job id (LSF not SLURM) which is appended to the file name.

You can use also batch job scripts for both of these forms. The script for the first example may be (exjob2.sh)

#!/bin/sh
#BSUB -n 4
#BSUB -o program.out
#BSUB -M 1048576
srun program

It is submitted by the command:

bsub < exjob2.sh

The script (exjob3.sh) for the second example would be:

#!/bin/sh
#BSUB -n 4
#BSUB -o prog.o.%J
#BSUB -M 1048576
#BSUB -ext "SLURM[nodes=4]"
srun prog

It is submitted byt the command

bsub < exjob3.sh

How to kill, stop and resume a batch job

The following shows how to check what jobs are running, including threir job IDs, and  how to kill one job (bkill).

user1@c553:/wrk/user1> bjobs
JOBID USER  STAT QUEUE    FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
11636 user1 PEND parallel c553              *un program May 15 07:27
user1@c553:/wrk/user1> bkill 11636
Job <11636> is being terminated

If it would have not killed it might have stopped by the command

bstop 11636

Its execution can be resumed:

bresume 11636

and its output file would have  been:

user1@c553:/wrk/user1> ls -la mpi_prog.out.11636 
-rw-------  1 user1 group2 1088 May 15 07:27 mpi_prog.out.11636