===== Slurm ===== ==== Job Submission ==== Slurm jobs can be submitted through a [[https://portal.darwin.bioeng.ru/pun/sys/myjobs|Job Composer]] app inside Slurm tab. This provides a graphical alternative to submitting jobs from the command line or from environments such as JupyterLab. Job script examples can be taken from [[https://portal.darwin.bioeng.ru/pun/sys/myjobs/workflows/new|Templates]]. The controller does not requeue batch jobs by default (%%JobRequeue=0%%). To allow a job to be requeued after a node failure or preemption, add //%%--%%requeue// to its script or submission command. Slurm starts a requeued batch script from the beginning with the same job ID, so jobs that must continue from prior work need application-level checkpointing and restart logic. The site GROMACS template already includes //%%--%%requeue// and //%%--%%qos preemptible// and checks for a checkpoint; these options are intended only for restart-safe runs. ==== Preemption ==== To allow other users to interrupt your long job (e.g., MD simulation) when they need resources, submit it with //%%--%%qos=preemptible// and //%%--%%requeue//. With %%PreemptMode=REQUEUE%%, an eligible preemptible job is requeued when a higher-priority job in the %%preempting%% QoS needs its resources. If you need resources sooner for a short task (e.g., neural network training), use //%%--%%qos=preempting//. Requeued jobs wait for available resources and start their batch script again from the beginning. **Ensure the job supports safe restarts (e.g. uses checkpoints) before submitting with %%--%%qos=preemptible --- otherwise it will restart from scratch after being requeued!** There is no way to enforce that the preempting job is short, so use this feature responsibly. ==== Job Time Limit Extension ==== If your job is likely to hit the time limit before completing, [[..:support|contact an administrator]] to have it extended. ==== GRES ==== Available GPU models for "%%--%%gres=gpu::": | GPU model | VRAM | | nvidia_geforce_rtx_2080_ti | 11 %%GB%% | | nvidia_geforce_rtx_3070_ti | 8 %%GB%% | | nvidia_geforce_rtx_3080 | 10 %%GB%% | | nvidia_rtx_a4500 | 20 %%GB%% | | nvidia_rtx_a5000 | 24 %%GB%% | | nvidia_rtx_a6000 | 48 %%GB%% | ==== Partitions ==== Each user is assigned to exactly one partition based on their Unix group (**users**/**intbio**/**struct_biotech**/**aigen**). Below are default parameters and limits for corresponding Slurm partitions: == users (a.k.a. guests) == * Nodes: 9–12 * Allowed groups: users, students * Max CPUs per node: 8 * Max memory per node: 16G * Max nodes: 1 * Default time: 08:00:00 * Max time: 08:00:00 == intbio == * Nodes: 1--6, 9--14 * Allowed groups: intbio * Default time: 08:00:00 * Max time: 30-00:00:00 == struct_biotech == * Nodes: 7--8 * Allowed groups: struct_biotech * Default time: 08:00:00 == aigen == * Nodes: 1--6, 9--14 * Allowed groups: aigen * Default time: 08:00:00 ==== Resource Visibility ==== Unlike Newton, Darwin enforces strict resource isolation via Slurm cgroups. Limits are not advisory. Users cannot access resources that were not requested (CPU, memory, GPU). GPU devices outside the allocation are not visible (e.g., via nvidia-smi).