===== Darwin Management =====
==== Node Management ====
New nodes can be added with Luna by cloning any existing node and then adjusting their IP and MAC addresses.
Nodes can also be managed through groups and inherit group parameters unless they are overridden manually.
After any hardware reconfiguration, refresh the hardware inventory to avoid stale health alerts (or in AlertX %%GUI%% ---> Reset Nodes):
alertx hardware reset -H node01.cluster,node02.cluster,node03.cluster,node04.cluster,node05.cluster,node06.cluster,node07.cluster,node08.cluster,node09.cluster,node10.cluster,node11.cluster,node12.cluster,node13.cluster,node14.cluster,storage-node01.cluster
Exports are managed by trinityX playbooks, but they can still be adjusted manually in ///etc/exports.d/trinity.exports// when necessary.
==== BMC ====
Passwords for the head-node and compute-node BMCs are stored in //secrets.txt//. To change a BMC password manually, run:
/opt/IPMICFG-Linux.x86_64 -user setpswd 2
==== Slurm ====
Most Slurm configuration lives in ///etc/slurm//, mainly in the following files:
* ///etc/slurm/slurm.conf//
* ///etc/slurm/slurm-nodes.conf//
* ///etc/slurm/slurm-partitions.conf//
* ///etc/slurm/gres.conf//
* ///etc/slurm/cgroup.conf//
* ///etc/slurm/cgroup_allowed_devices_file.conf//
UNIX users are synchronized with Slurm Quality of Service (QoS) accounting every minute via:
* ///usr/local/sbin/slurm-sync-users//
* ///etc/default/slurm-sync-users//
* ///etc/systemd/system/slurm-sync-users.timer//
* ///etc/systemd/system/slurm-sync-users.service//
==== Grafana ====
Darwin's Grafana has a few custom mechanisms worth knowing about:
* UNIX-to-Grafana user synchronization. The Authenticated Users team allows guests to read only the main page, while non-admin users can read all dashboards. Relevant files:
* ///usr/local/sbin/grafana-sync-authenticated-team//
* ///etc/systemd/system/grafana-sync-authenticated-team.service//
* ///etc/systemd/system/grafana-sync-authenticated-team.timer//
* Slurm textfile collection for the Slurm dashboard:
* ///usr/local/bin/slurm-job-names-collector.sh//
* Per-user block I/O textfile collection for System / Storage Disk (intbio, structbio, scratch):
* ///usr/local/bin/user-blockio-collector.py//
* ///etc/systemd/system/user-blockio-collector.service//
* ///etc/systemd/system/user-blockio-collector.timer//
* On storage-node01 the same script is used with TEXTFILE_DIR=///var/lib/prometheus_node_exporter// (packed in raid0.tar.gz / storage-image).
* Per-user disk-quota textfile collection for System / Storage Disk (intbio, structbio, opt, scratch, cryostorage). The timer runs //repquota// every 30 seconds and writes gauge //node_user_quota_used_bytes//:
* ///usr/local/bin/user-quota-collector.py//
* ///etc/systemd/system/user-quota-collector.service//
* ///etc/systemd/system/user-quota-collector.timer//
* On darwin the units are in grafana.tar.gz. The storage-image copies are in raid0.tar.gz. Only local ext4 mounts are exported, into ///trinity/local/var/lib/prometheus_node_exporter/user_quota.prom// on darwin and ///var/lib/prometheus_node_exporter/user_quota.prom// on storage-node01.
==== CryoSPARC ====
CryoSPARC is managed by systemctl (cryosparc.service).
Lanes are stored in the following directories:
* ///trinity/structbio_home/cryosparcuser/lanes/intbio//
* ///trinity/structbio_home/cryosparcuser/lanes/struct_biotech//
After changing a lane, re-apply it from the corresponding directory. Example:
cd /trinity/structbio_home/cryosparcuser/lanes/intbio
/trinity/structbio_home/cryosparcuser/cryosparc_master/bin/cryosparcm cluster connect
==== Disk Quotas ====
User and group quota accounting is enabled. Setting a limit does not require remounting: it takes effect immediately on the server that owns the filesystem. Example:
setquota -u v_petrov 9T 10T 0 0 /trinity/structbio_home
The four numbers are the soft and hard space limits, then the soft and hard file-count limits. ''0'' means no limit. The suffixes ''K'', ''M'', ''G'', and ''T'' mean kibibytes, mebibytes, gibibytes, and tebibytes. A soft limit can be exceeded for the grace period (currently 7 days); after that, writes are refused the same way as with a hard limit.
Use ''-g'' instead of ''-u'' to set a group limit. To check the result:
repquota -us /trinity/structbio_home
A limit applies to one filesystem only. Set limits for //%%/trinity/home%%//, //%%/trinity/structbio_home%%//, //%%/opt%%// on darwin, and for //%%/mnt/scratch%%//, //%%/mnt/cryostorage%%// on storage-node01.