How to use persistent storage in Singularity


Singularity containers are immutable read-only by default. Overlays allow you to put on top of the container a writable file system.

Singularity automatically binds inside the image the $HOME folder of the user and the folder where the .sif is, which means any changes made to that folder are permanent. You will not be able to write/execute/delete in all other paths inside a .sif container. This means that just by running bash inside the container you cannot make changes to the filesystem.

There are two ways to utilize persistence inside the container:

  • overlay image
  • writable sandbox directories

Overlay image

Creating an overlay image

The overlay image is used as storage. Therefore we first need a file system image. Use the command below to create 500MB of storage.

dd if=/dev/zero of=overlay.img bs=1M count=500 && \
	mkfs.ext3 overlay.img

Then we can start a container with any of the above commands described in the previous section and provide the overlay option to be used.

singularity shell --overlay overlay.img lent_latest.sif
singularity run -o overlay.img lent_latest.sif foamVersion
singularity exec -o overlay.img lent_latest.sif python ./

Managing the overlay image

This image later can be moved around and it contains only the changes made to the file system. That means that if we run the container without the overlay image then we will not see any changes we made before.

Copy the image to/from the Lichtenberg cluster
scp . # From cluster to CWD
scp overlay.sif # From CWD to cluser

This image can be added (mounted) to the local file system and the changes only be viewed, which is useful for storing simulation results.


Expanding the storage size of the image
# Change overlay.img to 700MB from 500MB (previous example command)
e2fsck -f overlay.img && \
	resize2fs my_overlay 700M

2. Creating writable sandbox directories

Here is a way to a container within a writable directory (called a sandbox). It’s possible to create a sandbox without root privileges, but to ensure proper file permissions it is recommended to do so as root. The resulting directory operates just like a container in a SIF file. To make changes within the container, use the --writable flag when you invoke your container. Note that the permissions for the user inside the container are the same as the ones outside of it (on the host system).

sudo singularity build --sandbox <NAME_OF_IMAGE> <DEF_FILE>/<LINK_TO_DOCKER_REPO>

You can convert a mutable container to an immutable one normally with the build command and vice versa

sudo singularity build lent_immutable.sif lent_mutable/

The mutable folder will be considerable larger in size than the compressed single .sif file and there also might be issues while transfering the folder around because of permission rights.

See also