How to run Jupyter notebooks on the (Lichtenberg) cluster and use them remotely

Instead of pulling the aggregated data from the Lichtenberg cluster all the time, the same Jupyter notebook that is used to visualize and document the simulations/tests locally, can be used on the cluster.

This is based on the post Use a jupyter notebook remotely. I found that a password needs to be set for the jupyter notebooks on the remote machine so these steps are added.

Starting and connecting to Jupyter notebooks on the (Lichtenberg) cluster

Steps

  1. Configure jupyter notebook as a regular user.
  2. Set the password for the jupyter notebook.
  3. Start the jupyter notebook without the browser.
  4. Map the ports.
  5. Connect to the jupyter notebook from a remote machine.

The steps are described here for the Lichtenberg cluster using the login node naming provided there, but they work for any machine that provides SSH access.

1. Configure jupyter notebook as a regular user.

On the cluster, execute the following command to generate a default configuration file:

$ jupyter notebook --generate-config

Writing default config to: /work/home/TUID/.jupyter/jupyter_notebook_config.py

2. Set the passwort for the jupyter notebook.

Execute the following command and enter and confirm the password for the notebooks on the remote machine:

$ jupyter notebook password

3. Start the jupyter notebook without the browser.

Leave this terminal open. Not sure if this runs when using screen, just access the machine in the command line terminal regularly and execute this:

$ jupyter notebook --no-browser --port=8889

4. Map the ports

On the local machine, execute:

$ ssh -N -f -L localhost:8888:localhost:8889 username@your_remote_host_name

5. Connect to the jupyter notebook from a remote machine.

In your web browser, on the local machine, enter

localhost:8888

Additional information

Securing a notebook server

Use a jupyter notebook remotely

Make sure you use the cluster user name (TUID at TU Darmstadt) and the proper hostname of the login node where you started the jupyter notebook on the cluster. Clusters have multiple login nodes so starting a jupyter notebook on one node, and listening on another in step 5 makes no sense.

See also