Singularity Containers

For more information see: https://sylabs.io/docs

Basic Idea

The basic idea behind Singularity containers is that software is packaged into a container (basically an entire self-contained operating system in a file!) that can be based on a Docker image that can then be run by the user. This allows hard to install software to be easily packaged and used - because you are packaging the entire OS!

During runtime, the root file system / is changed to the one inside the image and file systems are brought into the container through bind mounts. Effectively, the programs in the container are run in an environment mostly defined by the container image, but the programs can read and write specific files in Triton. Typically, e.g. the home directory comes from Triton.

This sounds complicated, but in practice this is easy due to singularity_wrapper written for Triton. You can also run singularity on triton without the wrapper, but you may need to e.g. bind /scratch yourself to access your data.

Basic Usage with singularity_wrapper

While the image itself is read-only, remember that /home, /m, /scratch and /l etc. are not. If you edit/remove files in these locations within the image, that will happen outside the image as well.

On Triton, you just need to load the proper module. This will set some environment variables and enable the use of singularity_wrapper.

singularity_wrapper is written so that when you load a module written for a singularity image, all the important options are already handled for you. It has three basic commands:

  1. singularity_wrapper shell <shell> - Gives user a shell within the image (specify <shell> to say which shell you want).

  2. singularity_wrapper exec <cmd> - Executes a program within the image.

  3. singularity_wrapper run <parameters> - Runs the singularity image. What this means depends on the image in question - each image will define a “run command” which does something. If you don’t know what this is, use the first two instead.

Under the hood, singularity_wrapper does this:

  1. Choosing appropriate image based on module version

  2. Binding of basic paths (-B /l:/l, /m:/m, /scratch:/scratch)

  3. Loading of system libraries within images (if needed) (e.g. -B /lib64/nvidia:/opt/nvidia)

  4. Setting working directory within image (if needed)

Short guide to Singularity commands

These are the “raw” singularity commands. If you use these, you have to configure the images and bind mounts yourself (which is done automatically by singularity_wrapper). If you module show the module you can get hints about what happens.

Singularity enables three base commands to user:

  1. singularity shell <image> - Gives user a shell within the image (see singularity shell --help for more information on flags etc.)

  2. singularity exec <image> <cmd> - Executes a program within the image (see singularity exec --help for more information on flags etc.)

  3. singularity run <image> <parameters> - Runs the singularity image. What this means depends on the image in question. (see singularity run --help for more information on flags etc.)

An example use case with MPI in singularity

The Serpent code is a Hybrid MPI/OpenMP particle following code, and can be installed into a container using the definition file sss2.def, which creates a container based on Ubuntu v. 20.04. In the build process, Singularity clones the Serpent source code, installs the required compilers and libraries, including the MPI library to the container. Furthermore, datafiles needed by Serpent are included in the container. Finally, a python environment with useful tools are also installed into the container. The Serpent code is compiled and the executable binaries are saved and the source code is removed.

The container can be directly used with the Triton queue system assuming the datafiles are stored in the user home folder. The file sss2.slurm_cmd can be used as an example. If scratch is used, please add -B /scratch after “exec” in the file.

The key observations to make:

  1. mpirun is called in Triton, which launches multiple Singularity containers (one for each MPI task). Each container directly launches the `sss2`-executable. Each container can run multiple OpenMP threads of Serpent.

  2. The openMPI library (v. 4.0.3) shipping with Ubuntu 20.04 seems to be compatible with the Triton module openmpi/4.0.5

  3. The Ubuntu MPI library binds all the threads to the same CPU. This is avoided by passing the parameter --bind-to none to mpirun.

  4. The infiniband is made available by the mpirun parameter --mca btl_openib_allow_ib.