New nodes can be added with Luna by cloning any existing node and then adjusting their IP and MAC addresses.

Nodes can also be managed through groups and inherit group parameters unless they are overridden manually.

After any hardware reconfiguration, refresh the hardware inventory to avoid stale health alerts:

alertx hardware reset -H node01.cluster,node02.cluster,node03.cluster,node04.cluster,node05.cluster,node06.cluster,node07.cluster,node08.cluster,node09.cluster,node10.cluster,node11.cluster,node12.cluster,node13.cluster,node14.cluster

Exports are managed by trinityX playbooks, but they can still be adjusted manually in /etc/exports.d/trinity.exports when necessary.

Passwords for the head-node and compute-node BMCs are stored in secrets.txt. To change a BMC password manually, run:

/opt/IPMICFG-Linux.x86_64 -user setpswd 2 <PASSWORD>

Most Slurm configuration lives in /etc/slurm, mainly in the following files:

• /etc/slurm/slurm.conf
• /etc/slurm/slurm-nodes.conf
• /etc/slurm/slurm-partitions.conf
• /etc/slurm/gres.conf
• /etc/slurm/cgroup.conf
• /etc/slurm/cgroup_allowed_devices_file.conf

UNIX users are synchronized with Slurm Quality of Service (QoS) accounting every minute via:

• /usr/local/sbin/slurm-sync-users
• /etc/default/slurm-sync-users
• /etc/systemd/system/slurm-sync-users.timer
• /etc/systemd/system/slurm-sync-users.service

Darwin's Grafana has two custom mechanisms worth knowing about:

• UNIX-to-Grafana user synchronization. The Authenticated Users team allows guests to read only the main page, while non-admin users can read all dashboards. Relevant files:
  • /usr/local/sbin/grafana-sync-authenticated-team
  • /etc/systemd/system/grafana-sync-authenticated-team.service
  • /etc/systemd/system/grafana-sync-authenticated-team.timer
• Slurm textfile collection for the Slurm dashboard:
  • /usr/local/bin/slurm-job-names-collector.sh

CryoSPARC lanes are stored in the following directories:

• /trinity/structbio_home/cryosparcuser/lanes/intbio
• /trinity/structbio_home/cryosparcuser/lanes/struct_biotech

After changing a lane, re-apply it from the corresponding directory. Example:

cd /trinity/structbio_home/cryosparcuser/lanes/intbio
/trinity/structbio_home/cryosparcuser/cryosparc_master/bin/cryosparcm cluster connect

1. Enter the image:

lchroot compute-image

2. Install necessary packages:

dnf install -y nano

3. Exit the image:

exit

4. Pack the image:

luna osimage pack compute-image

5. Reboot the compute/storage nodes:

luna control power reset \
    node01,node02,node03,node04,node05,node06,node07,node08,node09,node10,node11,node12,node13,node14

6. Backup the image after the new configuration is tested successfully:

luna osimage remove full-compute-image-bak
luna osimage clone -o compute-image full-compute-image-bak

Five main images are maintained:

• raw-compute-image-bak: a fresh Rocky 9.6 compute-node image after running ansible-playbook compute-default.yml
• compute-image: the working image for compute nodes
• full-compute-image-bak: a backup copy of compute-image
• storage-image: the working image for storage nodes
• full-storage-image-bak: a backup copy of storage-image

Hot changes can be applied in bulk with:

pdsh -w node[01-14] "dnf install -y nano"

Spack can manage custom executables. The example below creates a simple hello-world package:

• Create an executable

  mkdir -p /tmp/dummy-bin &&
  
  cat > /tmp/dummy-bin/hello-world << EOF
  #!/bin/bash
  echo "Hello from my custom binary managed by Spack!"
  EOF
  
  chmod +x /tmp/dummy-bin/hello-world

• Pack the executable

  tar \
     -czf /tmp/dummy-software-1.0.tar.gz \
     -C /tmp/dummy-bin \
     hello-world

• Create a local Spack repository

  mkdir -p /opt/spack/repos &&
  
  spack repo create \
     /opt/spack/repos/local_repo \
     local_repo &&
  
  spack repo add \
     /opt/spack/repos/local_repo/spack_repo/local_repo

• Copy the archive hash

  sha256sum /tmp/dummy-software-1.0.tar.gz

• Create a package file

  spack create \
     --name dummy-software \
     file:///tmp/dummy-software-1.0.tar.gz

Insert the hash into the generated version line, for example version('1.0.0', sha256='<HASH>').

• Install the package

  spack install dummy-software &&
  
  module load Core/dummy-software &&
  
  hello-world

• Configurations and overrides can be backed up in encrypted form (see secrets.txt for the password) to Bioengineering Department NextCloud:

  dir=~/Documents/bioeng_data/darwin_assets
  assets=/root/assets
  
  ssh darwin_root "
      mkdir -p '$assets'
      tar -czpf '$assets'/ood.tar.gz \
          /var/www/ood/public/custom.css \
          /usr/share/ondemand-dex/web/themes/ondemand/styles.css \
          /etc/ood/config/ondir=~/Documents/bioeng_data/darwin_assets
  assets=/root/assets
  
  ssh darwin_root "
      mkdir -p '$assets'
      tar -czpf '$assets'/ood.tar.gz \
          /var/www/ood/public/custom.css \
          /usr/share/ondemand-dex/web/themes/ondemand/styles.css \
          /etc/ood/config/ondemand.d/ondemand.yml \
          /etc/ood/config/nginx_stage.yml \
          2>/dev/null
      tar -czpf '$assets'/apps.tar.gz \
          /var/www/ood/apps/sys \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/passwd/templates/index.html \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/app.py \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/js/custom.js \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/templates/osusers_modal.html \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/base/css/custom.css \
          /etc/ood/config/apps \
          /etc/selinux/local \
          2>/dev/null
      tar -czpf '$assets'/nginx.tar.gz \
          /etc/nginx \
          2>/dev/null
      tar -czpf '$assets'/letsencrypt.tar.gz \
          /etc/letsencrypt \
          2>/dev/null
      tar -czpf '$assets'/dokuwiki.tar.gz \
          /var/www/dokuwiki \
          2>/dev/null
      tar -czpf '$assets'/grafana.tar.gz \
          /etc/grafana/grafana.ini \
          /var/lib/grafana \
          /usr/local/bin/slurm-job-names-collector.sh \
          ​​/usr/local/sbin/grafana-sync-authenticated-team \
          /etc/systemd/system/grafana-sync-authenticated-team.service \
          /etc/systemd/system/grafana-sync-authenticated-team.timer \
          2>/dev/null
      tar -czpf '$assets'/slurm.tar.gz \
          /trinity/shared/etc/slurm \
          /usr/local/sbin/slurm-sync-users \
          /etc/default/slurm-sync-users \
          /etc/systemd/system/slurm-sync-users.service \
          /etc/systemd/system/slurm-sync-users.timer \
          2>/dev/null
      tar -czpf '$assets'/nut.tar.gz \
          /etc/nut \
          /etc/systemd/system/prometheus-nut-exporter.service \
          /trinity/local/etc/prometheus_server/file_sd/other/nut_exporter.yml \
          2>/dev/null
      tar -czpf '$assets'/cryosparc.tar.gz \
          /etc/systemd/system/cryosparc.service \
          /trinity/structbio_home/cryosparcuser/lanes \
          /trinity/structbio_home/cryosparcuser/cryosparc_worker/config.sh \
          2>/dev/null
      tar -czpf '$assets'/raid0.tar.gz \
          /trinity/images/compute-image/usr/local/sbin/mount-scratch-raid.sh \
          /trinity/images/compute-image/etc/systemd/system/mount-scratch-raid.service \
          2>/dev/null
      tar -czpf '$assets'/netconsole.tar.gz \
          /etc/logrotate.d/netconsole \
          /usr/local/sbin/netconsole-receiver \
          /etc/rsyslog.d/netconsole-6666.conf \
          /etc/systemd/system/netconsole-receiver.service \
          2>/dev/null
  "
  
  scp -q \
      darwin_root:"$assets"/ood.tar.gz \
      darwin_root:"$assets"/apps.tar.gz \
      darwin_root:"$assets"/letsencrypt.tar.gz \
      darwin_root:"$assets"/nginx.tar.gz \
      darwin_root:"$assets"/dokuwiki.tar.gz \
      darwin_root:"$assets"/grafana.tar.gz \
      darwin_root:"$assets"/slurm.tar.gz \
      darwin_root:"$assets"/nut.tar.gz \
      darwin_root:"$assets"/cryosparc.tar.gz \
      darwin_root:"$assets"/raid0.tar.gz \
      darwin_root:"$assets"/netconsole.tar.gz \
      "$dir"
  
  for f in "$dir"/*.tar.gz; do
      gpg --batch --yes \
          --symmetric \
          --cipher-algo AES256 \
          --compress-algo none \
          --passphrase-file ~/.darwin_backup_password \
          "$f" &&
      rm "$f"
  done
  
  bak_dir="$dir"_baks
  n=$(ls "$bak_dir"/darwin_assets_bak*.tar.gz 2>/dev/null \
      | sed -E 's/.*bak([0-9]+)\.tar.gz/\1/' \
      | sort -n | tail -1)
  n=$(( ${n:-0} + 1 ))
  tar -czpf "$bak_dir/darwin_assets_bak$n.tar.gz" \
      -C "$(dirname "$dir")" "$(basename "$dir")"
  demand.d/ondemand.yml \
          /etc/ood/config/nginx_stage.yml \
          2>/dev/null
      tar -czpf '$assets'/apps.tar.gz \
          /var/www/ood/apps/sys \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/passwd/templates/index.html \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/app.py \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/js/custom.js \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/templates/osusers_modal.html \
          /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/base/css/custom.css \
          /etc/ood/config/apps \
          /etc/selinux/local \
          2>/dev/null
      tar -czpf '$assets'/nginx.tar.gz \
          /etc/nginx \
          2>/dev/null
      tar -czpf '$assets'/letsencrypt.tar.gz \
          /etc/letsencrypt \
          2>/dev/null
      tar -czpf '$assets'/dokuwiki.tar.gz \
          /var/www/dokuwiki \
          2>/dev/null
      tar -czpf '$assets'/grafana.tar.gz \
          /etc/grafana/grafana.ini \
          /var/lib/grafana \
          /usr/local/bin/slurm-job-names-collector.sh \
          ​​/usr/local/sbin/grafana-sync-authenticated-team \
          /etc/systemd/system/grafana-sync-authenticated-team.service \
          /etc/systemd/system/grafana-sync-authenticated-team.timer \
          2>/dev/null
      tar -czpf '$assets'/slurm.tar.gz \
          /trinity/shared/etc/slurm \
          /usr/local/sbin/slurm-sync-users \
          /etc/default/slurm-sync-users \
          /etc/systemd/system/slurm-sync-users.service \
          /etc/systemd/system/slurm-sync-users.timer \
          2>/dev/null
      tar -czpf '$assets'/nut.tar.gz \
          /etc/nut \
          /etc/systemd/system/prometheus-nut-exporter.service \
          /trinity/local/etc/prometheus_server/file_sd/other/nut_exporter.yml \
          2>/dev/null
      tar -czpf '$assets'/cryosparc.tar.gz \
          /etc/systemd/system/cryosparc.service \
          /trinity/structbio_home/cryosparcuser/lanes \
          /trinity/structbio_home/cryosparcuser/cryosparc_worker/config.sh \
          2>/dev/null
      tar -czpf '$assets'/raid0.tar.gz \
          /trinity/images/compute-image/usr/local/sbin/mount-scratch-raid.sh \
          /trinity/images/compute-image/etc/systemd/system/mount-scratch-raid.service \
          2>/dev/null
      tar -czpf '$assets'/netconsole.tar.gz \
          /etc/logrotate.d/netconsole \
          /usr/local/sbin/netconsole-receiver \
          /etc/rsyslog.d/netconsole-6666.conf \
          /etc/systemd/system/netconsole-receiver.service \
          2>/dev/null
  "
  
  scp -q \
      darwin_root:"$assets"/ood.tar.gz \
      darwin_root:"$assets"/apps.tar.gz \
      darwin_root:"$assets"/letsencrypt.tar.gz \
      darwin_root:"$assets"/nginx.tar.gz \
      darwin_root:"$assets"/dokuwiki.tar.gz \
      darwin_root:"$assets"/grafana.tar.gz \
      darwin_root:"$assets"/slurm.tar.gz \
      darwin_root:"$assets"/nut.tar.gz \
      darwin_root:"$assets"/cryosparc.tar.gz \
      darwin_root:"$assets"/raid0.tar.gz \
      darwin_root:"$assets"/netconsole.tar.gz \
      "$dir"
  
  for f in "$dir"/*.tar.gz; do
      gpg --batch --yes \
          --symmetric \
          --cipher-algo AES256 \
          --compress-algo none \
          --passphrase-file ~/.darwin_backup_password \
          "$f" &&
      rm "$f"
  done
  
  bak_dir="$dir"_baks
  n=$(ls "$bak_dir"/darwin_assets_bak*.tar.gz 2>/dev/null \
      | sed -E 's/.*bak([0-9]+)\.tar.gz/\1/' \
      | sort -n | tail -1)
  n=$(( ${n:-0} + 1 ))
  tar -czpf "$bak_dir/darwin_assets_bak$n.tar.gz" \
      -C "$(dirname "$dir")" "$(basename "$dir")"

To unpack a backup archive:

tar -xzpf darwin_assets_bak1.tar.gz

To decrypt archived files (the password is in secrets.txt):

for f in darwin_assets/*.tar.gz.gpg; do
    gpg --batch \
        --passphrase-file ~/.darwin_backup_password \
        --decrypt \
        "$f" > "${f%.gpg}"
done