You want to submit several jobs that have dependencies, to create a workflow. E.g., Job B will only start after Job A is finished. Job C will only start after Job B is finished. And so on.
SLURM provide powerful mechanism to define job dependencies, by the job submission command sbatch -d type:id. The keyword id could be job id or job array id. There are 4 available type :
afterany: This job could begin after specified jobs have terminated.
afterok: This job could begin after specified jobs have successfully terminated (e.g., exit code zero).
afternotok: This job could begin after specified jobs have failed.
after: This job could begin after specified jobs have begun. (e.g., if you use this keyword, and specify job2 depends on job1, job2 could start immediately once job1 starts!)
Let say you first submitted a job. You run this in terminal, and the job id shows in the screen.
sbatch job1.sh
Submitted batch job 123456
Then you have a second job. You want to make sure the second job could only start running after the first job is successfully terminated. You could run this in the terminal. Notice that in the real case, you have to replace 123456 to your actual job id.
sbatch -d afterok:123456 job2.sh
or
sbatch --dependency=afterok:123456 job2.sh
Alternatively, you could have the second job ready to start after the first job is terminated, whether sucessfully or not. The command will be:
sbatch -d afterany:123456 job2.sh
or
sbatch --dependency=afterany:123456 job2.sh
You could also specify multiple job depencencies on 1 job. E.g., if you want job3 starts only after job1 and job2 are successfully finished, the command will be:
sbatch -d afterok:<jobid-of-job1>:<jobid-of-job2> job3.sh
NYUAD HPC Apps Team:
--------------------
- Benoit Marchand
- Guowei He
- Jorge Naranjo
Please refer to the online documentation available here