How to use the geophase Singularity container

Singularity is a container software. It is basically similar to Docker .

For more information please refer to the literature.

Here are the instructions on how to use the Singularity container for the geophase project.

You can use the following commands with the .sif produced by the Definition file (geophase.def) at the end of the article.

  • clone : To get the project from the container to the current working directory.

    ./geophase.sif clone
    
  • build: By default the project in the /opt/geophase/ path will be built on a folder named “build” on the host current working directory.

    ./geophase.sif build
    

    To build a modified geophase project which lies on another path (e.g. ~/modified_geophase) provide the path as a second argument. This way the modified project will use the environment of the container to build inside that folder.

    ./geophase.sif build ~/modified_geophase/
    
  • test: By default the project in the /opt/geophase/ path will be tested on a folder named “test” on the host current working directory. To test a modified geophase project which lies on another path (e.g. ~/modified_geophase) provide the path as a second argument. This way the modified project will use the environment and the build of the container to test inside that folder.

    ./geophase.sif test ~/modified_geophase/
    

    If a project path is provided, that project has to have already been built inside a build/ folder.

  • getdata: So far getdata just copies all .csv .png and .vtk files from the build folder of the container to the host current working directory.

Embed an overlay into the geophase.sif container

In order to make use of the Jupyter notebook , it needs to have a writable filesystem available. For that we have to create an overlay images and embed it inside the read-only container.

To create an overlay:

dd if=/dev/zero of=overlay.img bs=1M count=500 && mkfs.ext3 overlay.img

Note: This will create a 500 MB overlay. Adjust by changing the block size (bs) and the count.

To embed the overlay into the container (named geophase.sif in this example):

singularity sif add --datatype 4 --partfs 2 --parttype 4 --partarch 2 --groupid 1 geophase.sif overlay.img

–datatype 4 means the type of data is Partition, –partfs 2 means the filesystem is Ext3, –parttype 4 means its Overlay partition, –partarch 2 means architecture is amd64 (x86-64).

To run the writable container:

singularity shell --writable geophase.sif

Now, you can create/delete/modify files inside the container, provided that you also have the permissions to do so and these changes are transfered along with the .sif.

To inspect the container:

This command will give you information about the filesystems of the container. Example output provided and we can see that at the end there is a filystem of type Overlay Ext3 which is the one we embedded.

singularity sif list geophase.sif
 
Container id: 275b9e9f-e6cc-4cb4-b737-299f8552da54
Created on:   2020-04-03 11:57:04 +0200 CEST
Modified on:  2020-04-03 15:16:45 +0200 CEST
----------------------------------------------------
Descriptor list:
ID   |GROUP   |LINK    |SIF POSITION (start-end)  |TYPE
------------------------------------------------------------------------------
1    |1       |NONE    |32768-36250               |Def.FILE
2    |1       |NONE    |36864-36984               |JSON.Generic
3    |1       |NONE    |40960-1126567936          |FS (Squashfs/*System/amd64)
4    |1       |NONE    |1126567936-1650855936     |FS (Ext3/Overlay/amd64)

To remove an embedded overlay:

singularity sif del 4 geophase.sif

The number after the del command indicates the ID of the object descriptor (in this case the Ext3-Overlay). Important: After this, everything that was saved in the overlay will be deleted and will not be retrievable.

To build the container use:

sudo singularity build geophase.sif geophase.def
Bootstrap: docker
From: archlinux:latest
 
%post
    pacman -Sy --noconfirm reflector
    reflector --latest 5 --sort rate --save /etc/pacman.d/mirrorlist
    pacman -Syu --noconfirm gcc cmake eigen python python-matplotlib texlive-core python-seaborn python-pandas jupyter make git png++ boost texlive-latexextra
    cd /opt/
    mkdir geophase
    git clone --branch feature/geom-vof-init https://git.rwth-aachen.de/leia/geophase.git geophase/
    cd geophase/
    mkdir build && cd build/
    cmake -DCMAKE_BUILD_TYPE=Release ..
    make
    chmod -R 777 /opt/geophase/
 
%runscript
    # CMD will carry the command the user provided.
    CMD=$1
 
    # WORKDIR will be pointed to the location where files will be built and/or tested.
    WORKDIR=/opt/geophase/
 
    # PROJDIR will be pointing to the location of the project.
    PROJDIR=`cd ${2:-$WORKDIR} && pwd`
 
    # Check whether user uses overlay or not.
    OVERLAY=`cat /proc/mounts | grep overlay | cut -d " " -f 4 | echo $(head -c 2)`
 
    if [ "$OVERLAY" = "rw" ]; then
        echo "You are using an Overlay. Everything will be saved in $(echo $WORKDIR)"
    elif [ "$OVERLAY" = "ro" ]; then
        WORKDIR=$(echo $PWD)
        echo "You are not using an Overlay. Everything will be saved in $(echo $WORKDIR)"
    else
        echo "Cannot determine if using an Overlay or not."
    fi
 
    # Move to the wokring directory. Either /opt/geophase/ if overlay is used, or the users PWD if not.
    cd $WORKDIR
 
    case $CMD in
        getdata) # Copy *.csv *.vtk *.png files to the $WORKDIR.
            echo "Copying data into folder data/"
            mkdir -p data/
            cp /opt/geophase/build/*.csv data/
            cp /opt/geophase/build/*.vtk data/
            cp /opt/geophase/build/*.png data/
            ;;
        build) # Create a new build folder and rebuild geophase project.
            echo "Building geophase($PROJDIR) into folder build/"
 
            # Check if a project directory was provided instead of the default one.
            if [ $# -eq 2 ]; then
                cd $PROJDIR
            fi
 
            # Check if a "build" folder already exists in the project directory.
            if [ -d "build" ]; then
                while true; do
                    read -p "A folder named "build" already exists in $WORKDIR. Do you want to overwrite it? " yn
                        case $yn in
                            [Yy]* ) rm -rf build/*; break;;
                            [Nn]* ) exit;;
                            * ) echo "Please answer with a Y/y or N/n.";;
                        esac
                done
            fi
 
            mkdir build/ && cd build/
            cmake -DCMAKE_BUILD_TYPE=Release $PROJDIR
            make
            ;;
        test) # Create a testing folder and retest the geophase binaries.
            echo "Testing geophase($PROJDIR) into folder tests/"
 
            # Check if a project directory was provided instead of the default one.
            if [ $# -eq 2 ]; then
                cd $PROJDIR
            fi
 
            mkdir -p tests/ && rm -rf tests/* && cd tests/
            cp $PROJDIR/build/CTestTestfile.cmake .
            ctest -V
            rm CTestTestfile.cmake
            ;;
        clone) # Copy the 'geophase' project to the CWD without the 'build' folder.
            echo "Copying the geophase source code to $(echo $WORKDIR)/geophase"
            mkdir -p geophase/
            cp -r `ls -Ad /opt/geophase/* | grep -v "build"` ./geophase/
            ;;
        jupyter)
            # Check if a project directory was provided instead of the default one.
            if [ $# -eq 2 ]; then
                cd $PROJDIR/
            fi
 
            cd jupyter-notebooks/
            jupyter notebook --no-browser --port=8889
    esac
 
%test
    cd /opt/geophase/build
    ctest -V
 
%help
    This container consists of the environment, the code, the build and the tests for the geophase project (https://git.rwth-aachen.de/leia/geophase).
 
    Usage:
        ./geophase.sif <command>
        singularity run geophase.sif <command>
        singularity shell [-o overlay.img] geophase.sif (use with overlay)
        singularity shell --writable geophase.sif (use with embedded overlay)
 
    The commands are:
        clone                        copies the geophase from /opt/geophase to the current working directory
        getdata                      copes all *.csv *.vtk *.png files from /opt/geophase/build to the current working directory
        build   [path-to-project]    builds the geophase project. By default, the '/opt/geophase'
        test    [path-to-project]    run the tests on the geophase project. By default, the '/opt/geophase/build'
        jupyter [path-to-project]    runs a Jupyter notebook. By default, the '/opt/geophase/jupyter-notebooks/geophase-interface-positioning.ipynb'

See also