How to upload all figures from a Jupyter notebook to a paper repository

Below is provided a shell script that is called with one single argument, the Jupyter notebook you wish to extract all the figures from.

#!/bin/bash
 
# Helper script to update figures on paper repository.
 
PAPER_REPO="https://git.rwth-aachen.de/john.doe/XXXXXXXX.git"
 
git config --global credential.helper store
git clone $PAPER_REPO paper_repo/
 
# Extract the images from the Notebook.
jupyter nbconvert --to markdown --output paper_figure.md $1
rm paper_figure.md
mkdir -p paper_repo/figures
mv paper_figure_files/* paper_repo/figures
rmdir paper_figure_files
 
cd paper_repo
if [ $(git config --global user.email) ]; then
    git add .
    git commit -m "Update figures"
    git push origin master
else
    echo "Unable to auto-detect email address. Run git config --global user.email 'you@example.com'"
fi
 
# Cleanup
cd ..
rm -rf paper_repo

Before you even run the script you need to modify the first line, which will point to the paper repository. The script is written in a way that it is assumed that the paper repository is private and that this script is executed on the Lichtenberg cluster where there git and jupyter are not installed. So in order for the script to work you need to work from the insides of a Singularity container where git and jupyter are installed.

The first time the script is run, the username and password are going to be requested, since the script uses https in order to pull / push the project. But after that first time (unless you clean the git store cache), the username/password are not going to be asked again. Furthermore, the user needs to provide an email to the git (git config --global user.email 'you@example.com').

git store will save the credentials as a plaintext file inside your home folder.

  1. The script first clones the repository inside the folder where the script is.
  2. Then, the jupyter nbconvert command will convert the .ipynb file to a Markdown file (.md) where in the process it will also create a folder with all the images of the notebook.
  3. After that, the script will delete the .md file since its useless and move the figures folder inside the cloned repository of the scientific paper.
  4. There is a check for the user.email which needs to be defined in order for the pull to succeed and if that is the case the script uses the stored credentials to push the commit . Otherwise, the user is prompted with a message in order to provide the email (only the first time).

The notebook has to be already executed in some way so the figures are available already.

See also