administration_guides:darwin

Darwin Management

Darwin Management

Node Management

New nodes can be added with Luna by cloning any existing node and then adjusting their IP and MAC addresses.

Nodes can also be managed through groups and inherit group parameters unless they are overridden manually.

After any hardware reconfiguration, refresh the hardware inventory to avoid stale health alerts:

alertx hardware reset -H node01.cluster,node02.cluster,node03.cluster,node04.cluster,node05.cluster,node06.cluster,node07.cluster,node08.cluster,node09.cluster,node10.cluster,node11.cluster,node12.cluster,node13.cluster,node14.cluster

Exports are managed by trinityX playbooks, but they can still be adjusted manually in /etc/exports.d/trinity.exports when necessary.

BMC

Passwords for the head-node and compute-node BMCs are stored in secrets.txt. To change a BMC password manually, run:

/opt/IPMICFG-Linux.x86_64 -user setpswd 2 <PASSWORD>

Slurm

Most Slurm configuration lives in /etc/slurm, mainly in the following files:

/etc/slurm/slurm.conf
/etc/slurm/slurm-nodes.conf
/etc/slurm/slurm-partitions.conf
/etc/slurm/gres.conf
/etc/slurm/cgroup.conf
/etc/slurm/cgroup_allowed_devices_file.conf

UNIX users are synchronized with Slurm Quality of Service (QoS) accounting every minute via:

/usr/local/sbin/slurm-sync-users
/etc/default/slurm-sync-users
/etc/systemd/system/slurm-sync-users.timer
/etc/systemd/system/slurm-sync-users.service

Grafana

Darwin's Grafana has two custom mechanisms worth knowing about:

UNIX-to-Grafana user synchronization. The Authenticated Users team allows guests to read only the main page, while non-admin users can read all dashboards. Relevant files:
- /usr/local/sbin/grafana-sync-authenticated-team
- /etc/systemd/system/grafana-sync-authenticated-team.service
- /etc/systemd/system/grafana-sync-authenticated-team.timer
Slurm textfile collection for the Slurm dashboard:
- /usr/local/bin/slurm-job-names-collector.sh

CryoSPARC

CryoSPARC lanes are stored in the following directories:

/trinity/structbio_home/cryosparcuser/lanes/intbio
/trinity/structbio_home/cryosparcuser/lanes/struct_biotech

After changing a lane, re-apply it from the corresponding directory. Example:

cd /trinity/structbio_home/cryosparcuser/lanes/intbio
/trinity/structbio_home/cryosparcuser/cryosparc_master/bin/cryosparcm cluster connect

Installation of Packages on Compute Nodes

1. Enter the image:

lchroot compute-image

2. Install necessary packages:

dnf install -y nano

3. Exit the image:

exit

4. Pack the image:

luna osimage pack compute-image

5. Reboot the compute/storage nodes:

luna control power reset \
    node01,node02,node03,node04,node05,node06,node07,node08,node09,node10,node11,node12,node13,node14

6. Backup the image after the new configuration is tested successfully:

luna osimage remove full-compute-image-bak
luna osimage clone -o compute-image full-compute-image-bak

Five main images are maintained:

raw-compute-image-bak: a fresh Rocky 9.6 compute-node image after running ansible-playbook compute-default.yml
compute-image: the working image for compute nodes
full-compute-image-bak: a backup copy of compute-image
storage-image: the working image for storage nodes
full-storage-image-bak: a backup copy of storage-image

Hot changes can be applied in bulk with:

pdsh -w node[01-14] "dnf install -y nano"

Custom Spack Packages

Spack can manage custom executables. The example below creates a simple hello-world package:

Create an executable

mkdir -p /tmp/dummy-bin &&

cat > /tmp/dummy-bin/hello-world << EOF
#!/bin/bash
echo "Hello from my custom binary managed by Spack!"
EOF

chmod +x /tmp/dummy-bin/hello-world

Pack the executable

tar \
   -czf /tmp/dummy-software-1.0.tar.gz \
   -C /tmp/dummy-bin \
   hello-world

Create a local Spack repository

mkdir -p /opt/spack/repos &&

spack repo create \
   /opt/spack/repos/local_repo \
   local_repo &&

spack repo add \
   /opt/spack/repos/local_repo/spack_repo/local_repo

Copy the archive hash

sha256sum /tmp/dummy-software-1.0.tar.gz

Create a package file

spack create \
   --name dummy-software \
   file:///tmp/dummy-software-1.0.tar.gz

Insert the hash into the generated version line, for example version('1.0.0', sha256='<HASH>').

Install the package

spack install dummy-software &&

module load Core/dummy-software &&

hello-world

Backups

Configurations and overrides can be backed up in encrypted form (see secrets.txt for the password) to Bioengineering Department NextCloud:

dir=~/Documents/bioeng_data/darwin_assets
assets=/root/assets

ssh darwin_root "
    mkdir -p '$assets'
    tar -czpf '$assets'/ood.tar.gz \
        /var/www/ood/public/custom.css \
        /usr/share/ondemand-dex/web/themes/ondemand/styles.css \
        /etc/ood/config/ondir=~/Documents/bioeng_data/darwin_assets
assets=/root/assets

ssh darwin_root "
    mkdir -p '$assets'
    tar -czpf '$assets'/ood.tar.gz \
        /var/www/ood/public/custom.css \
        /usr/share/ondemand-dex/web/themes/ondemand/styles.css \
        /etc/ood/config/ondemand.d/ondemand.yml \
        /etc/ood/config/nginx_stage.yml \
        2>/dev/null
    tar -czpf '$assets'/apps.tar.gz \
        /var/www/ood/apps/sys \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/passwd/templates/index.html \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/app.py \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/js/custom.js \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/templates/osusers_modal.html \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/base/css/custom.css \
        /etc/ood/config/apps \
        /etc/selinux/local \
        2>/dev/null
    tar -czpf '$assets'/nginx.tar.gz \
        /etc/nginx \
        2>/dev/null
    tar -czpf '$assets'/letsencrypt.tar.gz \
        /etc/letsencrypt \
        2>/dev/null
    tar -czpf '$assets'/dokuwiki.tar.gz \
        /var/www/dokuwiki \
        2>/dev/null
    tar -czpf '$assets'/grafana.tar.gz \
        /etc/grafana/grafana.ini \
        /var/lib/grafana \
        /usr/local/bin/slurm-job-names-collector.sh \
        /usr/local/sbin/grafana-sync-authenticated-team \
        /etc/systemd/system/grafana-sync-authenticated-team.service \
        /etc/systemd/system/grafana-sync-authenticated-team.timer \
        2>/dev/null
    tar -czpf '$assets'/slurm.tar.gz \
        /trinity/shared/etc/slurm \
        /usr/local/sbin/slurm-sync-users \
        /etc/default/slurm-sync-users \
        /etc/systemd/system/slurm-sync-users.service \
        /etc/systemd/system/slurm-sync-users.timer \
        2>/dev/null
    tar -czpf '$assets'/nut.tar.gz \
        /etc/nut \
        /etc/systemd/system/prometheus-nut-exporter.service \
        /trinity/local/etc/prometheus_server/file_sd/other/nut_exporter.yml \
        2>/dev/null
    tar -czpf '$assets'/cryosparc.tar.gz \
        /etc/systemd/system/cryosparc.service \
        /trinity/structbio_home/cryosparcuser/lanes \
        /trinity/structbio_home/cryosparcuser/cryosparc_worker/config.sh \
        2>/dev/null
    tar -czpf '$assets'/raid0.tar.gz \
        /trinity/images/compute-image/usr/local/sbin/mount-scratch-raid.sh \
        /trinity/images/compute-image/etc/systemd/system/mount-scratch-raid.service \
        2>/dev/null
    tar -czpf '$assets'/netconsole.tar.gz \
        /etc/logrotate.d/netconsole \
        /usr/local/sbin/netconsole-receiver \
        /etc/rsyslog.d/netconsole-6666.conf \
        /etc/systemd/system/netconsole-receiver.service \
        2>/dev/null
"

scp -q \
    darwin_root:"$assets"/ood.tar.gz \
    darwin_root:"$assets"/apps.tar.gz \
    darwin_root:"$assets"/letsencrypt.tar.gz \
    darwin_root:"$assets"/nginx.tar.gz \
    darwin_root:"$assets"/dokuwiki.tar.gz \
    darwin_root:"$assets"/grafana.tar.gz \
    darwin_root:"$assets"/slurm.tar.gz \
    darwin_root:"$assets"/nut.tar.gz \
    darwin_root:"$assets"/cryosparc.tar.gz \
    darwin_root:"$assets"/raid0.tar.gz \
    darwin_root:"$assets"/netconsole.tar.gz \
    "$dir"

for f in "$dir"/*.tar.gz; do
    gpg --batch --yes \
        --symmetric \
        --cipher-algo AES256 \
        --compress-algo none \
        --passphrase-file ~/.darwin_backup_password \
        "$f" &&
    rm "$f"
done

bak_dir="$dir"_baks
n=$(ls "$bak_dir"/darwin_assets_bak*.tar.gz 2>/dev/null \
    | sed -E 's/.*bak([0-9]+)\.tar.gz/\1/' \
    | sort -n | tail -1)
n=$(( ${n:-0} + 1 ))
tar -czpf "$bak_dir/darwin_assets_bak$n.tar.gz" \
    -C "$(dirname "$dir")" "$(basename "$dir")"
demand.d/ondemand.yml \
        /etc/ood/config/nginx_stage.yml \
        2>/dev/null
    tar -czpf '$assets'/apps.tar.gz \
        /var/www/ood/apps/sys \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/passwd/templates/index.html \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/app.py \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/js/custom.js \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/templates/osusers_modal.html \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/base/css/custom.css \
        /etc/ood/config/apps \
        /etc/selinux/local \
        2>/dev/null
    tar -czpf '$assets'/nginx.tar.gz \
        /etc/nginx \
        2>/dev/null
    tar -czpf '$assets'/letsencrypt.tar.gz \
        /etc/letsencrypt \
        2>/dev/null
    tar -czpf '$assets'/dokuwiki.tar.gz \
        /var/www/dokuwiki \
        2>/dev/null
    tar -czpf '$assets'/grafana.tar.gz \
        /etc/grafana/grafana.ini \
        /var/lib/grafana \
        /usr/local/bin/slurm-job-names-collector.sh \
        /usr/local/sbin/grafana-sync-authenticated-team \
        /etc/systemd/system/grafana-sync-authenticated-team.service \
        /etc/systemd/system/grafana-sync-authenticated-team.timer \
        2>/dev/null
    tar -czpf '$assets'/slurm.tar.gz \
        /trinity/shared/etc/slurm \
        /usr/local/sbin/slurm-sync-users \
        /etc/default/slurm-sync-users \
        /etc/systemd/system/slurm-sync-users.service \
        /etc/systemd/system/slurm-sync-users.timer \
        2>/dev/null
    tar -czpf '$assets'/nut.tar.gz \
        /etc/nut \
        /etc/systemd/system/prometheus-nut-exporter.service \
        /trinity/local/etc/prometheus_server/file_sd/other/nut_exporter.yml \
        2>/dev/null
    tar -czpf '$assets'/cryosparc.tar.gz \
        /etc/systemd/system/cryosparc.service \
        /trinity/structbio_home/cryosparcuser/lanes \
        /trinity/structbio_home/cryosparcuser/cryosparc_worker/config.sh \
        2>/dev/null
    tar -czpf '$assets'/raid0.tar.gz \
        /trinity/images/compute-image/usr/local/sbin/mount-scratch-raid.sh \
        /trinity/images/compute-image/etc/systemd/system/mount-scratch-raid.service \
        2>/dev/null
    tar -czpf '$assets'/netconsole.tar.gz \
        /etc/logrotate.d/netconsole \
        /usr/local/sbin/netconsole-receiver \
        /etc/rsyslog.d/netconsole-6666.conf \
        /etc/systemd/system/netconsole-receiver.service \
        2>/dev/null
"

scp -q \
    darwin_root:"$assets"/ood.tar.gz \
    darwin_root:"$assets"/apps.tar.gz \
    darwin_root:"$assets"/letsencrypt.tar.gz \
    darwin_root:"$assets"/nginx.tar.gz \
    darwin_root:"$assets"/dokuwiki.tar.gz \
    darwin_root:"$assets"/grafana.tar.gz \
    darwin_root:"$assets"/slurm.tar.gz \
    darwin_root:"$assets"/nut.tar.gz \
    darwin_root:"$assets"/cryosparc.tar.gz \
    darwin_root:"$assets"/raid0.tar.gz \
    darwin_root:"$assets"/netconsole.tar.gz \
    "$dir"

for f in "$dir"/*.tar.gz; do
    gpg --batch --yes \
        --symmetric \
        --cipher-algo AES256 \
        --compress-algo none \
        --passphrase-file ~/.darwin_backup_password \
        "$f" &&
    rm "$f"
done

bak_dir="$dir"_baks
n=$(ls "$bak_dir"/darwin_assets_bak*.tar.gz 2>/dev/null \
    | sed -E 's/.*bak([0-9]+)\.tar.gz/\1/' \
    | sort -n | tail -1)
n=$(( ${n:-0} + 1 ))
tar -czpf "$bak_dir/darwin_assets_bak$n.tar.gz" \
    -C "$(dirname "$dir")" "$(basename "$dir")"

To unpack a backup archive:

tar -xzpf darwin_assets_bak1.tar.gz

To decrypt archived files (the password is in secrets.txt):

for f in darwin_assets/*.tar.gz.gpg; do
    gpg --batch \
        --passphrase-file ~/.darwin_backup_password \
        --decrypt \
        "$f" > "${f%.gpg}"
done

Table of Contents