A example batch script that uses IPython parallel (
within slurm. See also the interactive hints on the Python page.
ipyparallel uses global state in your home directory, so you can
only run _one_ of these at a time! You can add the
option to name different scripts (you could use
But then you will get a growing number of unneeded profile directories
~/.ipython/profile_*, so this isn’t recommended. Basically,
ipyparallel is more designed for one-at-a-time interactive use rather
than batch scripting (unless you do more work…).
ipyparallel.slrm is an example
slurm script that sets up ipyparallel. It assumes that most work is
done in the engines. It has inline Python, replace this with
#!/bin/bash #SBATCH -N 4 module load anaconda3 set -x ipcontroller --ip="*" & sleep 5 # Run the engines in slurm job steps (makes four of them, since we use # the -N slurm option)... srun ipengine --location=`hostname -f` & sleep 5 # Put the actual Python isn't in a job step. This is assuming that # most work happens in engines python3 <<EOF import os import ipyparallel client = ipyparallel.Client() result = client[:].apply_async(os.getpid) pid_map = result.get_dict() print(pid_map) EOF