IPython parallel
A example batch script that uses IPython parallel (ipyparallel
)
within slurm. See also the interactive hints on the Python page.
ipyparallel uses global state in your home directory, so you can
only run _one_ of these at a time! You can add the --profile=
option to name different scripts (you could use $SLURM_JOB_ID
).
But then you will get a growing number of unneeded profile directories
at ~/.ipython/profile_*
, so this isn’t recommended. Basically,
ipyparallel is more designed for one-at-a-time interactive use rather
than batch scripting (unless you do more work…).
ipyparallel.sh
is an example
slurm script that sets up ipyparallel. It assumes that most work is
done in the engines. It has inline Python, replace this with python
your_script_name.py
#!/bin/bash
#SBATCH --nodes=4
module load anaconda
set -x
ipcontroller --ip="*" &
sleep 5
# Run the engines in slurm job steps (makes four of them, since we use
# the --nodes=4 slurm option)...
srun ipengine --location=$(hostname -f) &
sleep 5
# Put the actual Python isn't in a job step. This is assuming that
# most work happens in engines
python3 <<EOF
import os
import ipyparallel
client = ipyparallel.Client()
result = client[:].apply_async(os.getpid)
pid_map = result.get_dict()
print(pid_map)
EOF