Batch Jobs With Slurm

Batch Jobs Batch jobs are jobs that are submitted via job script or commands that are input into the sbatch command interactively which will then enter the queueing system and prepare for the execution, then execute when possible. The execution could start immediately if the queue is not completely full, start after a short time period if preemption opted for, or after extensive time if the queue is full or running limits are already reached.

---

Batch Scripts

A simple sbatch script called 'hello.sh' to submit a simple "Hello World!" type problem follows:

#!/bin/bash
### Assume this file is named hello.sh
#SBATCH --account=projectname
echo "Hello World!"

The two '#SBATCH' directives above are required for all job submissions, whether interactive or batch. The values to account should be changed to the appropriate project account and the time should be changed to an appropriate walltime limit. This is a walltime limit, not CPU time. These values could also be supplied when submitting jobs by providing them directly on the command line when submitting. Slurm will default jobs to use one node, one task per node, and one cpu per node.

Submitting Jobs

When submitting batch jobs to the Slurm scheduler, use the 'sbatch' command along with a job script that contains all of the Slurm directives and desired commands to execute as the argument.

sbatch hello.sh

---

Single Node, Multi-Core Jobs

Slurm creates allocations of resources, and resources can vary depending on the work needing to be done with the cluster. A batch job that requires multiple cores can have a few different layouts depending on what is intending to be run. If the job is a multi-threaded application such as OpenMP or utilizes pthreads, it's best to set the number of tasks to 1. The below script will request that a single node with 4 cores available. The job script, assuming OpenMP, sets the number of threads to the job provided environment variable SLURM_CPUS_PER_TASK.

#!/bin/bash
#SBATCH --account=projectname
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=4
export $OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun 'your application'

Single Node, Multi-Tasks

This could be a multi-tasked job where the application has it's own parallel processing engine or uses MPI, but experiences poor scaling over multiple nodes.

#!/bin/bash
#SBATCH --account=projectname
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
### Assuming MPI application
srun 'your application'

Multi-Node, Non-Multithreaded

An application that strictly uses MPI often can use multiple nodes. However, there is often a chance that MPI type programs do not implement multithreading capability. Therefore, the number of cpus per task should be set to a value of 1.

#!/bin/bash
#SBATCH --account=projectname
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
### Assuming 'application' is on your $PATH environment variable
srun 'your application'

Multi-Node, Multithreaded

Some applications have been developed to take advantage of both distributed memory parallelism and shared memory parallelism such that they're capable of using MPI and threading together. This often requires the user to find the right balance based on additional resources required such as memory per task, network bandwidth, and node core count. The below example request that 4 nodes be allocated, each supporting 4 MPI ranks and each MPI rank supporting 4 threads. The total CPU request count aggregates to 64 (i.e., 4 x 4 x 4).

#!/bin/bash
#SBATCH --account=projectname
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=4
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun 'your application' -arg1 -arg2

---

Anatomy of a Typical Job

Below is an example job script with common commands. What follows is a breakdown of each line and corresponding directives within the script:

    #!/bin/bash
    ## Add required Slurm directives
    #SBATCH --account=projectname
    ## Additional Slurm directives as needed:
    #SBATCH --<option>=X
    ## Add email notifcation
    #SBATCH --mail-user=cowboyjoe@uwyo.edu
    #SBATCH --mail-type=BEGIN,END,FAIL
    ## Export any envrionment variables needed
    export VAR1
    ## Purge and load any required software modules
    module purge
    module load software1 software2 
    ## Execute code/scripts
    software ./softwarescript.xx