Running HPC Jobs with Slurm
At ARCC, we use the SLURM job scheduler for HPC cluster/resource management, and job scheduling. SLURM is responsible for allocating resources to users, and provides a framework for users to start, execute and monitor work on requested and allocated resources. It also allows users to schedule work for execution at a later time. There are differences in the Slurm settings on MedicineBow and AI4WY. Please see the documentation for each system to see those differences.
Jobs
A job is an allocation of resources like compute nodes, GPUs, or cores that get assigned to a user for a specific amount of time. Jobs may be interactive or submitted as a batch script for a later scheduled execution.
When a job gets assigned to a specific set of hardware (this may be a collection of nodes, cores, GPUs, etc.) the job can specify commands to initiate parallel work in the form of job steps based on an configuration within their allocated hardware.
-
Slurm Doumentation
Slurm has extensive documentation that they maintain. We recommend taking a look at what they have published along with what we describe in this section of the documentation.
-
Common Slurm Directives
Slurm has a number of directives that determine the behavior of jobs on ARCC's HPC systems.
-
Slurm Job Types
With Slurm compute jobs can be scheduled and run in a variety of ways. With background batch, interactive command-line & applications, or with array jobs.
-
Job Exit Codes
When Slurm jobs fail, often times there will be an exit code associated with it that provide insights on what went wrong so researchers can update their scripts.
Slurm Commands
sacct
Query detailed information about the job that has completed. Use this utility to get information about running or completed jobs
salloc
Request an interactive job for debugging and/or interactive computing. ARCC configures the salloc command to launch an interactive shell on individual compute nodes with your current environment carried over from the current session. This command requires specifying a project account (-A or --account=) and walltime (-t or --time=).
sbatch
Submit a batch job consisting of a single job or job array. Several methods can be used to submit batch jobs. A script file can be used and provided as an argument on the command line. Alternatively, and rarer, the use of standard input can be used and the batch job can be created interactively. We recommend writing the batch job in a script so that it may be referenced at a later time.
scancel
Cancel jobs after submission. Works on pending and running jobs. By default, provide a jobid or set of jobids to cancel. Alternatively, one can use sets of flags to cancel specific jobs relating to the account, name, partition, qos, reservation, nodelist. To cancel all array tasks, specify the parent jobid.
sinfo
View the status of the Slurm partitions or nodes. Status of nodes that are drained can be seen using the -R flag.
squeue
View what is running or waiting to run in the job queue. Several modifiers and formats can be supplied to the command. You may be interested in the use of arccq as an alternative. The command arccjobs also provides a summary.
srun
A front-end launcher for job steps which includes serial and parallel jobs. srun can be considered an equivalent to mpirun or mpiexec when launching MPI jobs. Using srun inside a job is defined to be a job step that provides accounting information relating to memory, cpu time, and other parameters that are valuable when a job terminates unexpectedly or historical information is needed.
seff
Shows detailed data on used resources in an easy-to-read format, but can only show one job at a time.
sreport
Obtain information regarding usage since the last database roll up (usually around midnight each day). sreport can be used as an interactive tool to see the usage of the clusters.
Info
There are some additional commands, however, they'll not be mentioned here because they're not that useful on our system for general users. It's important to note that reading the man pages on the Slurm commands can be highly beneficial and if you have questions, ARCC encourages you to request information on submitting jobs to arcc-help@uwyo.edu.
ARCC Specific Commands & Queries
arccjobs
Get a report of jobs currently running on the cluster arccjobs shows a summary of jobs, cpu resources, and requested/used cpu time. It doesn't take any arguments or options.
arccquota
Get a report of your common HPC data storage locations and usage. arccquota shows information relating to storage quotas. By default, this will display $HOME and $SCRATCH quotas first, followed by the user's associated project quotas. This is a change on Teton from Mount Moran, but the tool is much more comprehensive. The command takes arguments to do project-only (i.e., no $HOME or $SCRATCH info displayed), extensive listing of users' quotas and usage within project directories, can summarize quotas (i.e., no user-specific usage on project spaces).
showjob
Get job parameters, and details for a job. Running showjob provides parameters specified when the job was requested, details about the job including ID, start and end times, nodes, cores, exit codes, state, the working directory, and fairshare information for the user and their associated projects.
Preemption & Investments
Some ARCC HPC Systems have preemption enabled. Preemption is the act of stopping one or more low-priority jobs to let a high-priority job run. This is due to the fact that individual researchers have contributed investments to help fund expansions to the system. In return, ARCC allows these investor accounts to preempt any jobs that may be running on the investment they contributed to the system.
If you have submitted a job that has been allocated to a compute node that is part of another account’s investment then your job could be preempted and returned to the the Slurm queue. The preempted job may restart right away if there are available resources, if there are not, the job will stay in the queue until the investor account's job finishes. If a PI is interested in making an investment, please reach out to ARCC
If you do not want your jobs to be preempted, then within your submission script request the non-investor partition which contains nodes not part of any investment.