How to build a fvc-reconstruct Singularity container

Content

fvc-reconstruct is a project of the SFB 1194 from Tomislav Maric and Tobias Tolle.
It is a solver for OpenFOAM, that reconstructs “cell-centered velocity fields from scalar fluxes on general unstructured meshes.”

Set-up of the fvc-container

The definition file is broken down in 3 main sections:

  1. First we need to download OpenFOAM (version 19.12) and build it inside the Singularity container . So we download the dependencies for OpenFOAM first (git, openmpi, gcc , make and flex), clone the repository into /opt/openfoam/OpenFOAM-v1912/, source the etc/bashrc file and use wmake to build it.

  2. Second step is to install all the necessary dependencies for the fvc-reconstruct project (python, python-pip, texlive-core, texlive-latexextra, pyfoam, sympy, matplotlib, pandas, and jupyter) and then clone the fvc-reconstruct project..

  3. Finally, in the %post section we need to build the project, run the study and test the results. Before building we change the building folder by setting the $WM_PROJECT_USER_DIR variable of the bashrc file to point to /opt/fvc-reconstruct/build, instead of $HOME/OpenFOAM, which in this case would actually be /root/OpenFOAM/ because the singularity build command is run as sudo. After that we can safely use wmake to build foamTestFvcReconstruct and studyRun to get the .csv results.

  4. The test of the results is in the %test section, but it could also be in the %post section. Just because the %test section is not using the same environment as the one in %post we need to add some directories to the path in order to find the installed binaries (/bin:/sbin:/usr/sbin) and then we are ready to run the test. Please remember that the tests are inside the Jupyter notebook , so we just execute all the cells of the notebook and replace the .ipynb inplace. If any errors occured during the execution of the tests (which are at the end of the notebook) the process would stop.

singularity shell [-o overlay] fvc-reconstruct

you can also use the run it simply as a binary with the following 6 commands. But before the commands take place there are some checks and assignments general for all of them.

  • ’$CMD’ is used to store the command provided.
  • ’$WORKDIR’ is used to store the working directory (where things is going to be saved).
  • ’$PROJDIR’ is used to store the project directory (which project to build/run/test).
    If a second argument is provided to the binary it should be a directory path containing a cloned fvc-reconstruct project.
  • ’$BASHRC’ is used to store the bashrc file used, as we need to modify it if we clone the project. Then follows a check to see if an overlay is being used (see /proc/mounts) and if there is not, we change the ‘$WORKDIR’ to our current folder (on the host machine).
    If a fvc-project directory is provided as a second argument the ‘$BASHRC’ would point to that bashrc file.

clone

This command will first check if an overlay is being used and only if there is not it will copy the whole fvc-reconstruct project folder from /opt to the current working directory. Also, it will copy the bashrc file from /opt/openfoam/OpenFOAM-v1912/etc/ and modify it so that if you want to build this project the binary is saved inside this directory and not $HOME or the hard-coded /opt/fvc-reconstruct/build. Furthermore, the clone command will delete binaries, python caches and any other data (studyClean) in order to provide a clean state for the project.

./fvc-reconstruct-container.sif clone # Will copy to the current working directory

It does not matter whether you are using an overlay or not. This command will copy from /opt to the current working directory.

getdata

This command will only work when not using an overlay and will copy the whole data folder from /opt/fvc-reconstruct to the current working directory.

./fvc-reconstruct-container.sif getdata # Will copy to the current working directory

It does not matter whether you are using an overlay or not. This command will copy from /opt to the current working directory.

build

This command will first check if there is an overlay being used simultaneously with a project directory or if you are trying to build without an overlay and at the same time not providing a project directory. After those checks, it uses source $BASHRC to set up the environment variables necessary for OpenFOAM and wmake to actually build the binary.

Remember that if a directory is provided $BASHRC will point to the $PROJDIR/bashrc and the output of the command will be saved inside $PROJDIR/build/platforms/linux64GccDPInt32Opt/bin/

./fvc-reconstruct-container.sif build fvc-reconstruct              # To be used on project inside 'fvc-reconstruct' directory (host machine)
singularity run -o overlay.img fvc-reconstruct-container.sif build # To be used with overlay only

studyrun

This command again will make the same checks as the build command so that it informs the user he cannot execute anything inside /opt/fvc-reconstruct without an overlay the same way he cannot build outside without providing a project directory as a second argument. $BASHRC is being sourced, in order to get the $PROJDIR’s foamTestFvcReconstruct binary in the $PATH and then studyRun is issued to produce the results.

./fvc-reconstruct-container.sif studyrun fvc-reconstruct              # To be used on project inside 'fvc-reconstruct' directory (host machine)
singularity run -o overlay.img fvc-reconstruct-container.sif studyrun # To be used with overlay only

You first need to build the foamTestFvcReconstruct binary before trying to run the study. There is no checking of its existence and the command will fail if it cannot find it.

test

This command tests the results produced by the studyrun command. A simple execution of the cells of notebook fvc-reconstruct-convergence.ipynb is taking place creating 2 new .csv files inside the $PROJDIR/data folder.

./fvc-reconstruct-container.sif test fvc-reconstruct              # To be used on project inside 'fvc-reconstruct' directory (host machine)
singularity run -o overlay.img fvc-reconstruct-container.sif test # To be used with overlay only

You first need to run the study before trying to test the results.

jupyter

This command will move to the $PROJDIR/data folder and run jupyter on the notebook making it available as a web application on localhost:8889.

./fvc-reconstruct-container.sif jupyter fvc-reconstruct              # To be used on project inside 'fvc-reconstruct' directory (host machine)
singularity run -o overlay.img fvc-reconstruct-container.sif jupyter # To be used with overlay only

There is no password setting for the access of the notebook.

The Definition File

Here is a single file containing all the steps. Please consider breaking it up into 2 or more definition files (where OpenFOAM and/or fvc-reconstruct is already build in the base image to be used as base) in order to modify the commands so that building times are faster.

Bootstrap: docker
From: archlinux:latest
 
%post
    # Install dependencies
    pacman -Suy --noconfirm git openmpi gcc make flex python python-pip texlive-core texlive-latexextra
    pip install --no-cache pyfoam sympy matplotlib pandas jupyter
 
    # Download OpenFOAM
    git clone --branch OpenFOAM-v1912.200129 https://develop.openfoam.com/Development/openfoam.git /opt/openfoam/OpenFOAM-v1912/
     
    # Move to directory
    cd /opt/openfoam/OpenFOAM-v1912/
 
    # Source OF. Hack by setting/unsetting e so it does no exit because of sub-shell `return 1`
    source etc/bashrc || true
 
    # Install OF
    ./Allwmake -j -s -q
 
    # Clone project repository
    git clone --branch feature/testing https://git.rwth-aachen.de/leia/fvc-reconstruct.git /opt/fvc-reconstruct
 
    # Build foamTestFvcReconstruct project
    sed -i 's#WM_PROJECT_USER_DIR=.*#WM_PROJECT_USER_DIR="/opt/fvc-reconstruct/build"#g' /opt/openfoam/OpenFOAM-v1912/etc/bashrc
    source /opt/openfoam/OpenFOAM-v1912/etc/bashrc || true
 
    cd /opt/fvc-reconstruct/code/foamTestFvcReconstruct
    wmake
 
    # Run study
    cd /opt/fvc-reconstruct/data/hex2D
    ./studyRun
 
    # Chnage permissions
    chmod -R 777 /opt/fvc-reconstruct
 
    # Clean pacman cache
    pacman -Scc --noconfirm
 
%test
    export PATH=$PATH:/bin:/sbin:/usr/sbin
    cd /opt/fvc-reconstruct/data
    jupyter nbconvert --to notebook --inplace --execute fvc-reconstruct-convergence.ipynb
    chmod -R 777 /opt/fvc-reconstruct/data
 
%runscript
    # Will carry the command the user provided.
    CMD=$1
 
    WORKDIR=$PWD
    PROJDIR=$(realpath ${2:-/opt/fvc-reconstruct})
 
    # Check whether user uses overlay or not.
    OVERLAY=$(grep overlay < /proc/mounts |  cut -d " " -f 4 | head -c 2)
 
    if [ "$CMD" != "clone" ] && [ "$CMD" != "getdata" ]; then
        if [ "$OVERLAY" = "ro" ] && [ $# -ne 2 ]; then
            echo "You need either an overlay or to provide an 'fvc-reconstruct' project folder."
            exit
        fi
         
        if [ "$OVERLAY" = "rw" ] && [ $# -eq 1 ]; then
            WORKDIR=$PROJDIR
            BASHRC=/opt/openfoam/OpenFOAM-v1912/etc/bashrc
            cd $WORKDIR
        fi
 
        if [ $# -eq 2 ]; then
            BASHRC=$PROJDIR/bashrc
        fi
    fi
 
    case $CMD in
        clone) # Copy the project to the specified destination.
            CLONEDIR=$WORKDIR/fvc-reconstruct
            mkdir -p $CLONEDIR
  
            echo "Copying the fvc-reconstruct($PROJDIR) project to $CLONEDIR"
 
            cp -r /opt/fvc-reconstruct/* /opt/openfoam/OpenFOAM-v1912/etc/bashrc $CLONEDIR
            rm -r $CLONEDIR/build $CLONEDIR/code/foamTestFvcReconstruct/Make/linux64GccDPInt32Opt/
            cd $CLONEDIR/data/
            rm -r __pycache__ *.csv
            cd $CLONEDIR/data/hex2D/ && ./studyClean
 
            sed -i '/projectDir\=\"\/opt\/openfoam\/OpenFOAM\-\$WM_PROJECT\_VERSION\"/s/^#//g' $CLONEDIR/bashrc
            sed -i "s#WM_PROJECT_USER_DIR=.*#WM_PROJECT_USER_DIR=$CLONEDIR/build#g" $CLONEDIR/bashrc
            ;;
        build) # Build the specified project.
            echo "Building fvc-reconstruct($PROJDIR)"          
            source $BASHRC
            wmake $PROJDIR/code/foamTestFvcReconstruct
            ;;
        studyrun)
            echo "Running study ($PROJDIR)"
            source $BASHRC
            cd $PROJDIR/data/hex2D
            ./studyRun
            ;;
        test)
            echo "Testing project ($PROJDIR)"
            jupyter nbconvert --to notebook --inplace --execute "$PROJDIR"/data/fvc-reconstruct-convergence.ipynb
            ;;
        getdata)
            echo "Copying data into folder $WORKDIR/data"
            cp -r /opt/fvc-reconstruct/data $WORKDIR
            ;;
        jupyter) # Check if a project directory was provided instead of the default one.
            cd $PROJDIR/data
            jupyter notebook -y --generate-config
            jupyter notebook --no-browser --port=8889
            ;;
    esac

See also