IPython parallel

A example batch script that uses IPython parallel (ipyparallel) within slurm. See also the interactive hints on the Python page.

ipyparallel uses global state in your home directory, so you can only run _one_ of these at a time! You can add the --profile= option to name different scripts (you could use $SLURM_JOB_ID). But then you will get a growing number of unneeded profile directories at ~/.ipython/profile_*, so this isn’t recommended. Basically, ipyparallel is more designed for one-at-a-time interactive use rather than batch scripting (unless you do more work…).

ipyparallel.sh is an example slurm script that sets up ipyparallel. It assumes that most work is done in the engines. It has inline Python, replace this with python your_script_name.py

#SBATCH --nodes=4

module load scicomp-python-env
set -x

ipcontroller --ip="*" &
sleep 5
# Run the engines in slurm job steps (makes four of them, since we use
# the --nodes=4 slurm option)...
srun ipengine --location=$(hostname -f) &

sleep 5
# Put the actual Python isn't in a job step.  This is assuming that
# most work happens in engines
python3 <<EOF
import os
import ipyparallel
client = ipyparallel.Client()
result = client[:].apply_async(os.getpid)
pid_map = result.get_dict()