New nodes can be added with Luna by cloning any existing node and then adjusting their IP and MAC addresses.
Nodes can also be managed through groups and inherit group parameters unless they are overridden manually.
After any hardware reconfiguration, refresh the hardware inventory to avoid stale health alerts (or in AlertX GUI —> Reset Nodes):
alertx hardware reset -H node01.cluster,node02.cluster,node03.cluster,node04.cluster,node05.cluster,node06.cluster,node07.cluster,node08.cluster,node09.cluster,node10.cluster,node11.cluster,node12.cluster,node13.cluster,node14.cluster,storage-node01.cluster
Exports are managed by trinityX playbooks, but they can still be adjusted manually in /etc/exports.d/trinity.exports when necessary.
Passwords for the head-node and compute-node BMCs are stored in secrets.txt. To change a BMC password manually, run:
/opt/IPMICFG-Linux.x86_64 -user setpswd 2 <PASSWORD>
Most Slurm configuration lives in /etc/slurm, mainly in the following files:
UNIX users are synchronized with Slurm Quality of Service (QoS) accounting every minute via:
Darwin's Grafana has a few custom mechanisms worth knowing about:
CryoSPARC is managed by systemctl (cryosparc.service).
Lanes are stored in the following directories:
After changing a lane, re-apply it from the corresponding directory. Example:
cd /trinity/structbio_home/cryosparcuser/lanes/intbio /trinity/structbio_home/cryosparcuser/cryosparc_master/bin/cryosparcm cluster connect
User and group quota accounting is enabled. Setting a limit does not require remounting: it takes effect immediately on the server that owns the filesystem. Example:
setquota -u v_petrov 9T 10T 0 0 /trinity/structbio_home
The four numbers are the soft and hard space limits, then the soft and hard file-count limits. 0 means no limit. The suffixes K, M, G, and T mean kibibytes, mebibytes, gibibytes, and tebibytes. A soft limit can be exceeded for the grace period (currently 7 days); after that, writes are refused the same way as with a hard limit.
Use -g instead of -u to set a group limit. To check the result:
repquota -us /trinity/structbio_home
A limit applies to one filesystem only. Set limits for /trinity/home, /trinity/structbio_home, /opt on darwin, and for /mnt/scratch, /mnt/cryostorage on storage-node01.