Darwin Management
Node Management
New nodes can be added with Luna by cloning any existing node and then adjusting their IP and MAC addresses.
Nodes can also be managed through groups and inherit group parameters unless they are overridden manually.
After any hardware reconfiguration, refresh the hardware inventory to avoid stale health alerts (or in AlertX GUI —> Reset Nodes):
alertx hardware reset -H node01.cluster,node02.cluster,node03.cluster,node04.cluster,node05.cluster,node06.cluster,node07.cluster,node08.cluster,node09.cluster,node10.cluster,node11.cluster,node12.cluster,node13.cluster,node14.cluster,storage-node01.cluster
Exports are managed by trinityX playbooks, but they can still be adjusted manually in /etc/exports.d/trinity.exports when necessary.
BMC
Passwords for the head-node and compute-node BMCs are stored in secrets.txt. To change a BMC password manually, run:
/opt/IPMICFG-Linux.x86_64 -user setpswd 2 <PASSWORD>
Slurm
Most Slurm configuration lives in /etc/slurm, mainly in the following files:
- /etc/slurm/slurm.conf
- /etc/slurm/slurm-nodes.conf
- /etc/slurm/slurm-partitions.conf
- /etc/slurm/gres.conf
- /etc/slurm/cgroup.conf
- /etc/slurm/cgroup_allowed_devices_file.conf
UNIX users are synchronized with Slurm Quality of Service (QoS) accounting every minute via:
- /usr/local/sbin/slurm-sync-users
- /etc/default/slurm-sync-users
- /etc/systemd/system/slurm-sync-users.timer
- /etc/systemd/system/slurm-sync-users.service
Grafana
Darwin's Grafana has a few custom mechanisms worth knowing about:
- UNIX-to-Grafana user synchronization. The Authenticated Users team allows guests to read only the main page, while non-admin users can read all dashboards. Relevant files:
- /usr/local/sbin/grafana-sync-authenticated-team
- /etc/systemd/system/grafana-sync-authenticated-team.service
- /etc/systemd/system/grafana-sync-authenticated-team.timer
- Slurm textfile collection for the Slurm dashboard:
- /usr/local/bin/slurm-job-names-collector.sh
- Per-user block I/O textfile collection for System / Storage Disk (intbio, structbio, scratch):
- /usr/local/bin/user-blockio-collector.py
- /etc/systemd/system/user-blockio-collector.service
- /etc/systemd/system/user-blockio-collector.timer
- On storage-node01 the same script is used with TEXTFILE_DIR=/var/lib/prometheus_node_exporter (packed in raid0.tar.gz / storage-image).
- Per-user disk-quota textfile collection for System / Storage Disk (intbio, structbio, opt, scratch, cryostorage). The timer runs repquota every 30 seconds and writes gauge node_user_quota_used_bytes:
- /usr/local/bin/user-quota-collector.py
- /etc/systemd/system/user-quota-collector.service
- /etc/systemd/system/user-quota-collector.timer
- On darwin the units are in grafana.tar.gz. The storage-image copies are in raid0.tar.gz. Only local ext4 mounts are exported, into /trinity/local/var/lib/prometheus_node_exporter/user_quota.prom on darwin and /var/lib/prometheus_node_exporter/user_quota.prom on storage-node01.
CryoSPARC
CryoSPARC is managed by systemctl (cryosparc.service).
Lanes are stored in the following directories:
- /trinity/structbio_home/cryosparcuser/lanes/intbio
- /trinity/structbio_home/cryosparcuser/lanes/struct_biotech
After changing a lane, re-apply it from the corresponding directory. Example:
cd /trinity/structbio_home/cryosparcuser/lanes/intbio /trinity/structbio_home/cryosparcuser/cryosparc_master/bin/cryosparcm cluster connect
Disk Quotas
User and group quota accounting is enabled. Setting a limit does not require remounting: it takes effect immediately on the server that owns the filesystem. Example:
setquota -u v_petrov 9T 10T 0 0 /trinity/structbio_home
The four numbers are the soft and hard space limits, then the soft and hard file-count limits. 0 means no limit. The suffixes K, M, G, and T mean kibibytes, mebibytes, gibibytes, and tebibytes. A soft limit can be exceeded for the grace period (currently 7 days); after that, writes are refused the same way as with a hard limit.
Use -g instead of -u to set a group limit. To check the result:
repquota -us /trinity/structbio_home
A limit applies to one filesystem only. Set limits for /trinity/home, /trinity/structbio_home, /opt on darwin, and for /mnt/scratch, /mnt/cryostorage on storage-node01.