slurmjobs 🐌

Automating Slurm job generation.

Generate a set of .sbatch files over a grid of parameters. A run script is created which will submit all generated jobs as once.

Now with slurmjobs.Singularity() support!

You can also use Shell which excludes the slurm/module references so you can test & run on your local machine or test server.

Installation

pip install slurmjobs

Usage

import slurmjobs

jobs = slurmjobs.Singularity(
   'python train.py', email='me@nyu.edu', n_gpus=2)

# generate jobs across parameter grid
run_script, job_paths = jobs.generate([
   # two different model types
   ('model', ['AVE', 'AVOL']),
   # try mono and stereo audio
   ('audio_channels', [1, 2]),
], epochs=500)

# NOTE:
#       extra keywords (e.g. ``epochs``) are also passed as arguments
#       across all scripts and are not included in the job name.

# ******************************
# ** everything was generated ** - now let's see the outputted paths.

slurmjobs.util.summary(run_script, job_paths)
Output:
Generated 4 job scripts:
	 jobs/train/train,model-AVE,audio_channels-1.sbatch
	 jobs/train/train,model-AVE,audio_channels-2.sbatch
	 jobs/train/train,model-AVOL,audio_channels-1.sbatch
	 jobs/train/train,model-AVOL,audio_channels-2.sbatch

To submit all jobs, run:
. jobs/train/run.sh

Getting Started:

API documentation

Indices and tables