===== Darwin Management =====

==== Node Management ====
New nodes can be added with Luna by cloning any existing node and then adjusting their IP and MAC addresses.

Nodes can also be managed through groups and inherit group parameters unless they are overridden manually.

After any hardware reconfiguration, refresh the hardware inventory to avoid stale health alerts: <code>
alertx hardware reset -H node01.cluster,node02.cluster,node03.cluster,node04.cluster,node05.cluster,node06.cluster,node07.cluster,node08.cluster,node09.cluster,node10.cluster,node11.cluster,node12.cluster,node13.cluster,node14.cluster
</code>

Exports are managed by trinityX playbooks, but they can still be adjusted manually in ///etc/exports.d/trinity.exports// when necessary.

==== BMC ====
Passwords for the head-node and compute-node BMCs are stored in //secrets.txt//. To change a BMC password manually, run: <code>
/opt/IPMICFG-Linux.x86_64 -user setpswd 2 <PASSWORD>
</code>

==== Slurm ====
Most Slurm configuration lives in ///etc/slurm//, mainly in the following files:
  * ///etc/slurm/slurm.conf//
  * ///etc/slurm/slurm-nodes.conf//
  * ///etc/slurm/slurm-partitions.conf//
  * ///etc/slurm/gres.conf//
  * ///etc/slurm/cgroup.conf//
  * ///etc/slurm/cgroup_allowed_devices_file.conf//

UNIX users are synchronized with Slurm Quality of Service (QoS) accounting every minute via:
  * ///usr/local/sbin/slurm-sync-users//
  * ///etc/default/slurm-sync-users//
  * ///etc/systemd/system/slurm-sync-users.timer//
  * ///etc/systemd/system/slurm-sync-users.service//

==== Grafana ====
Darwin's Grafana has two custom mechanisms worth knowing about:
  * UNIX-to-Grafana user synchronization. The Authenticated Users team allows guests to read only the main page, while non-admin users can read all dashboards. Relevant files:
    * ///usr/local/sbin/grafana-sync-authenticated-team//
    * ///etc/systemd/system/grafana-sync-authenticated-team.service//
    * ///etc/systemd/system/grafana-sync-authenticated-team.timer//
  * Slurm textfile collection for the Slurm dashboard:
    * ///usr/local/bin/slurm-job-names-collector.sh//

==== CryoSPARC ====
CryoSPARC lanes are stored in the following directories:
  * ///trinity/structbio_home/cryosparcuser/lanes/intbio//
  * ///trinity/structbio_home/cryosparcuser/lanes/struct_biotech//

After changing a lane, re-apply it from the corresponding directory. Example: <code>
cd /trinity/structbio_home/cryosparcuser/lanes/intbio
/trinity/structbio_home/cryosparcuser/cryosparc_master/bin/cryosparcm cluster connect
</code>

==== Installation of Packages on Compute Nodes ====

1. Enter the image:<code>
lchroot compute-image
</code>
2. Install necessary packages:<code>
dnf install -y nano
</code>
3. Exit the image:<code>
exit
</code>
4. Pack the image:<code>
luna osimage pack compute-image
</code>
5. Reboot the compute/storage nodes:<code>
luna control power reset \
    node01,node02,node03,node04,node05,node06,node07,node08,node09,node10,node11,node12,node13,node14
</code>
6. Backup the image after the new configuration is tested successfully:<code>
luna osimage remove full-compute-image-bak
luna osimage clone -o compute-image full-compute-image-bak
</code>

**Five main images are maintained:**
  * //raw-compute-image-bak//: a fresh Rocky 9.6 compute-node image after running //ansible-playbook compute-default.yml//
  * //compute-image//: the working image for compute nodes
  * //full-compute-image-bak//: a backup copy of //compute-image//
  * //storage-image//: the working image for storage nodes
  * //full-storage-image-bak//: a backup copy of //storage-image//

**Hot changes can be applied in bulk with:**<code>
pdsh -w node[01-14] "dnf install -y nano"
</code>

==== Custom Spack Packages ====
Spack can manage custom executables. The example below creates a simple //hello-world// package:
  * **Create an executable** <code>
mkdir -p /tmp/dummy-bin &&

cat > /tmp/dummy-bin/hello-world << EOF
#!/bin/bash
echo "Hello from my custom binary managed by Spack!"
EOF

chmod +x /tmp/dummy-bin/hello-world
</code>
  * **Pack the executable** <code>
tar \
   -czf /tmp/dummy-software-1.0.tar.gz \
   -C /tmp/dummy-bin \
   hello-world
</code>
  * **Create a local Spack repository** <code>
mkdir -p /opt/spack/repos &&

spack repo create \
   /opt/spack/repos/local_repo \
   local_repo &&

spack repo add \
   /opt/spack/repos/local_repo/spack_repo/local_repo
</code>
  * **Copy the archive hash** <code>
sha256sum /tmp/dummy-software-1.0.tar.gz
</code>
  * **Create a package file** <code>
spack create \
   --name dummy-software \
   file:///tmp/dummy-software-1.0.tar.gz
</code>

Insert the hash into the generated version line, for example //version('1.0.0', sha256='<HASH>')//.

  * **Install the package** <code>
spack install dummy-software &&

module load Core/dummy-software &&

hello-world
</code>

==== Backups ====
  * Configurations and overrides can be backed up in encrypted form (see //secrets.txt// for the password) to Bioengineering Department NextCloud: <code>
dir=~/Documents/bioeng_data/darwin_assets
assets=/root/assets

ssh darwin_root "
    mkdir -p '$assets'
    tar -czpf '$assets'/ood.tar.gz \
        /var/www/ood/public/custom.css \
        /usr/share/ondemand-dex/web/themes/ondemand/styles.css \
        /etc/ood/config/ondir=~/Documents/bioeng_data/darwin_assets
assets=/root/assets

ssh darwin_root "
    mkdir -p '$assets'
    tar -czpf '$assets'/ood.tar.gz \
        /var/www/ood/public/custom.css \
        /usr/share/ondemand-dex/web/themes/ondemand/styles.css \
        /etc/ood/config/ondemand.d/ondemand.yml \
        /etc/ood/config/nginx_stage.yml \
        2>/dev/null
    tar -czpf '$assets'/apps.tar.gz \
        /var/www/ood/apps/sys \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/passwd/templates/index.html \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/app.py \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/js/custom.js \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/templates/osusers_modal.html \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/base/css/custom.css \
        /etc/ood/config/apps \
        /etc/selinux/local \
        2>/dev/null
    tar -czpf '$assets'/nginx.tar.gz \
        /etc/nginx \
        2>/dev/null
    tar -czpf '$assets'/letsencrypt.tar.gz \
        /etc/letsencrypt \
        2>/dev/null
    tar -czpf '$assets'/dokuwiki.tar.gz \
        /var/www/dokuwiki \
        2>/dev/null
    tar -czpf '$assets'/grafana.tar.gz \
        /etc/grafana/grafana.ini \
        /var/lib/grafana \
        /usr/local/bin/slurm-job-names-collector.sh \
        ​​/usr/local/sbin/grafana-sync-authenticated-team \
        /etc/systemd/system/grafana-sync-authenticated-team.service \
        /etc/systemd/system/grafana-sync-authenticated-team.timer \
        2>/dev/null
    tar -czpf '$assets'/slurm.tar.gz \
        /trinity/shared/etc/slurm \
        /usr/local/sbin/slurm-sync-users \
        /etc/default/slurm-sync-users \
        /etc/systemd/system/slurm-sync-users.service \
        /etc/systemd/system/slurm-sync-users.timer \
        2>/dev/null
    tar -czpf '$assets'/nut.tar.gz \
        /etc/nut \
        /etc/systemd/system/prometheus-nut-exporter.service \
        /trinity/local/etc/prometheus_server/file_sd/other/nut_exporter.yml \
        2>/dev/null
    tar -czpf '$assets'/cryosparc.tar.gz \
        /etc/systemd/system/cryosparc.service \
        /trinity/structbio_home/cryosparcuser/lanes \
        /trinity/structbio_home/cryosparcuser/cryosparc_worker/config.sh \
        2>/dev/null
    tar -czpf '$assets'/raid0.tar.gz \
        /trinity/images/compute-image/usr/local/sbin/mount-scratch-raid.sh \
        /trinity/images/compute-image/etc/systemd/system/mount-scratch-raid.service \
        2>/dev/null
    tar -czpf '$assets'/netconsole.tar.gz \
        /etc/logrotate.d/netconsole \
        /usr/local/sbin/netconsole-receiver \
        /etc/rsyslog.d/netconsole-6666.conf \
        /etc/systemd/system/netconsole-receiver.service \
        2>/dev/null
"

scp -q \
    darwin_root:"$assets"/ood.tar.gz \
    darwin_root:"$assets"/apps.tar.gz \
    darwin_root:"$assets"/letsencrypt.tar.gz \
    darwin_root:"$assets"/nginx.tar.gz \
    darwin_root:"$assets"/dokuwiki.tar.gz \
    darwin_root:"$assets"/grafana.tar.gz \
    darwin_root:"$assets"/slurm.tar.gz \
    darwin_root:"$assets"/nut.tar.gz \
    darwin_root:"$assets"/cryosparc.tar.gz \
    darwin_root:"$assets"/raid0.tar.gz \
    darwin_root:"$assets"/netconsole.tar.gz \
    "$dir"

for f in "$dir"/*.tar.gz; do
    gpg --batch --yes \
        --symmetric \
        --cipher-algo AES256 \
        --compress-algo none \
        --passphrase-file ~/.darwin_backup_password \
        "$f" &&
    rm "$f"
done

bak_dir="$dir"_baks
n=$(ls "$bak_dir"/darwin_assets_bak*.tar.gz 2>/dev/null \
    | sed -E 's/.*bak([0-9]+)\.tar.gz/\1/' \
    | sort -n | tail -1)
n=$(( ${n:-0} + 1 ))
tar -czpf "$bak_dir/darwin_assets_bak$n.tar.gz" \
    -C "$(dirname "$dir")" "$(basename "$dir")"
demand.d/ondemand.yml \
        /etc/ood/config/nginx_stage.yml \
        2>/dev/null
    tar -czpf '$assets'/apps.tar.gz \
        /var/www/ood/apps/sys \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/passwd/templates/index.html \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/app.py \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/js/custom.js \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/templates/osusers_modal.html \
        /trinity/local/ondemand/trinityx-ood-2.0.1770215387/users/static/base/css/custom.css \
        /etc/ood/config/apps \
        /etc/selinux/local \
        2>/dev/null
    tar -czpf '$assets'/nginx.tar.gz \
        /etc/nginx \
        2>/dev/null
    tar -czpf '$assets'/letsencrypt.tar.gz \
        /etc/letsencrypt \
        2>/dev/null
    tar -czpf '$assets'/dokuwiki.tar.gz \
        /var/www/dokuwiki \
        2>/dev/null
    tar -czpf '$assets'/grafana.tar.gz \
        /etc/grafana/grafana.ini \
        /var/lib/grafana \
        /usr/local/bin/slurm-job-names-collector.sh \
        ​​/usr/local/sbin/grafana-sync-authenticated-team \
        /etc/systemd/system/grafana-sync-authenticated-team.service \
        /etc/systemd/system/grafana-sync-authenticated-team.timer \
        2>/dev/null
    tar -czpf '$assets'/slurm.tar.gz \
        /trinity/shared/etc/slurm \
        /usr/local/sbin/slurm-sync-users \
        /etc/default/slurm-sync-users \
        /etc/systemd/system/slurm-sync-users.service \
        /etc/systemd/system/slurm-sync-users.timer \
        2>/dev/null
    tar -czpf '$assets'/nut.tar.gz \
        /etc/nut \
        /etc/systemd/system/prometheus-nut-exporter.service \
        /trinity/local/etc/prometheus_server/file_sd/other/nut_exporter.yml \
        2>/dev/null
    tar -czpf '$assets'/cryosparc.tar.gz \
        /etc/systemd/system/cryosparc.service \
        /trinity/structbio_home/cryosparcuser/lanes \
        /trinity/structbio_home/cryosparcuser/cryosparc_worker/config.sh \
        2>/dev/null
    tar -czpf '$assets'/raid0.tar.gz \
        /trinity/images/compute-image/usr/local/sbin/mount-scratch-raid.sh \
        /trinity/images/compute-image/etc/systemd/system/mount-scratch-raid.service \
        2>/dev/null
    tar -czpf '$assets'/netconsole.tar.gz \
        /etc/logrotate.d/netconsole \
        /usr/local/sbin/netconsole-receiver \
        /etc/rsyslog.d/netconsole-6666.conf \
        /etc/systemd/system/netconsole-receiver.service \
        2>/dev/null
"

scp -q \
    darwin_root:"$assets"/ood.tar.gz \
    darwin_root:"$assets"/apps.tar.gz \
    darwin_root:"$assets"/letsencrypt.tar.gz \
    darwin_root:"$assets"/nginx.tar.gz \
    darwin_root:"$assets"/dokuwiki.tar.gz \
    darwin_root:"$assets"/grafana.tar.gz \
    darwin_root:"$assets"/slurm.tar.gz \
    darwin_root:"$assets"/nut.tar.gz \
    darwin_root:"$assets"/cryosparc.tar.gz \
    darwin_root:"$assets"/raid0.tar.gz \
    darwin_root:"$assets"/netconsole.tar.gz \
    "$dir"

for f in "$dir"/*.tar.gz; do
    gpg --batch --yes \
        --symmetric \
        --cipher-algo AES256 \
        --compress-algo none \
        --passphrase-file ~/.darwin_backup_password \
        "$f" &&
    rm "$f"
done

bak_dir="$dir"_baks
n=$(ls "$bak_dir"/darwin_assets_bak*.tar.gz 2>/dev/null \
    | sed -E 's/.*bak([0-9]+)\.tar.gz/\1/' \
    | sort -n | tail -1)
n=$(( ${n:-0} + 1 ))
tar -czpf "$bak_dir/darwin_assets_bak$n.tar.gz" \
    -C "$(dirname "$dir")" "$(basename "$dir")"
</code>

To unpack a backup archive: <code>
tar -xzpf darwin_assets_bak1.tar.gz
</code>

To decrypt archived files (the password is in //secrets.txt//): <code>
for f in darwin_assets/*.tar.gz.gpg; do
    gpg --batch \
        --passphrase-file ~/.darwin_backup_password \
        --decrypt \
        "$f" > "${f%.gpg}"
done
</code>