Migration from Newton

Consequences of the HPC renovation to be aware of

  • Home directories have moved from “/home” to “/trinity/home” — old hard-coded absolute paths will no longer work
  • Operating system changed from Ubuntu 18.04 to Rocky Linux 9.7 — binaries built on Newton may not run reliably
  • Workflow for running sbatch scripts has changed — jobs are now recommended to be submitted from the web portal rather than from JupyterLab

Steps to migrate

  • Update hard-coded absolute paths to start with “/trinity/home”. Where possible, avoid fixed paths and use portable alternatives (e.g., “~” in Bash or path utilities such as “os.path.abspath()” in Python)
  • Fix user-space Conda/pip environments (contact an administrator if shared environments need recreating), for that you can either:
    • Use VS Code. It enables a flexible way of finding kernels, so ipykernel reinstallation is not needed
    • Fix kernel path inside /trinity/home/<USER>/.conda/envs/<ENV>/share/jupyter/kernels/python3/kernel.json:
      “/home/<USER>/.conda/envs/<ENV>/bin/python” —> “/trinity/home/<USER>/.conda/envs/<ENV>/bin/python”
    • Reinstall ipykernel in your environment: 
      conda install -n <ENV> ipykernel -c conda-forge --force-reinstall -y

      for all envs: 

      for name in $(ls ~/.conda/envs);
do 
  conda install -n $name ipykernel -c conda-forge --force-reinstall -y; 
done
    • Recreate environment (if above does not work):
      • Export Conda environment to a file: 
        conda env export -n <OLD_ENV> --no-builds -f environment.yml
      • Create a new Conda environment from the file: 
        conda env create -n <NEW_ENV> -f environment.yml
      • In case the new environment does not work (e.g., cannot solve the environment/conflicts), repeat the first step, using: 
        conda env export -n <OLD_ENV> --from-history -f environment.yml
  • Rebuild manually compiled software and contact an administrator if centrally installed software needs to be rebuilt

Tips & Bonuses

  • JupyterLab now has a Python LSP server (jump to definitions and references, make use of dynamic tips)
  • JupyterLab now supports showing hidden files (can be enabled in JupyterLab settings)
  • JupyterLab now has a bunch of color themes
  • It is recommended to try VS Code (codeserver) instead of JupyterLabs. Advantages: a lot of features to be overwhelmed by. Disadvantages: a lot of features to be overwhelmed by