When running parameter variation , we want to programmatically adjust SLURM parameters (memory, number of processes, etc) depending on the parameters of the study (mesh density, etc). We don’t want the submission of the job to hang until the job has completed, we want to start everything in parallel.
For this to work without populating stdout with status reports about pending jobs, call(command, shell=True) is necessary. This means the command must be formulated not as a list, but as a single string. See this SO answer. Once the string is defined, at the end of the string redirection to /dev/null is done to supress output and the job is set to the background. The rest is taken care of by SLURM: resource allocation, start, running, everything runs as it was started with a SLURM script.
from subprocess import call
base_command="srun --mem-per-cpu=500 --time=00:10 --ntasks=1"
# Submit volume mesh generation to the SLURM workload manager.
variable_command=" --job-name %s -o %s.log %s >/dev/null 2>&1 &" % \
(args.mesh_generator, args.mesh_generator, args.mesh_generator)
call(base_command + variable_command, shell=True)
We can now adjust memory requirements and required time for execution depending on the mesh density.