A cluster becomes useful when it provides a dependable service to people, not when its nodes merely pass a hardware test. Define ownership, admission, software lifecycle, data protection and incident response before the first shared workload arrives.
Choose the operating model
Slurm is a natural fit for queued batch jobs, MPI and allocation-based research computing. Kubernetes is a natural fit for long-running services, APIs and declarative application operations. Some organisations run both with clear node or time boundaries, but competing control planes must never believe they own the same GPU simultaneously.
Write service objectives: supported hours, maintenance windows, queue policy, recovery targets and who can approve privileged workloads.
Make resources explicit
Schedulers should allocate whole GPUs or supported partitioning mechanisms, CPU cores, memory and local scratch together. Kubernetes discovers accelerators through device plugins; Slurm uses generic resources and cgroups. Prevent untracked interactive work from bypassing the scheduler because it defeats capacity planning and can corrupt performance results.
Fabric and storage operations
Monitor RDMA link state, errors, pause behaviour, retransmits and topology. Keep compute and storage networks logically distinct where the design provides both. GPUDirect and CUDA-aware MPI require a compatible end-to-end stack; include them in post-maintenance validation.
Set project quotas and scratch expiry. Back up authoritative datasets, configuration, identity data, code registries and experiment metadata; do not spend the same protection budget on reproducible temporary files.
Identity, isolation and supply chain
Integrate central identity, use least privilege and separate administrators from ordinary users. Protect container registries, sign images and restrict untrusted privileged containers. Segment management interfaces from user and production networks. Store secrets in a dedicated secrets system, not environment files in shared home directories.
For regulated or export-controlled work, map datasets and model artefacts to authorised users and destinations, keep auditable approvals and apply retention policies.
Observability and capacity management
Collect node health, temperatures, power, ECC events, CPU/RAM/GPU usage, GPU errors, filesystem capacity, queue wait and job outcomes. Prometheus can collect time-series metrics; scheduler accounting provides allocation history. Alert on service-impacting conditions, not every transient fluctuation.
Review demand by project and resource. A high queue does not always mean more GPUs: it may reveal memory, licence, storage or policy constraints.
Change and recovery drill
- Snapshot configuration and record firmware/driver versions.
- Drain workloads and establish a rollback point.
- Apply one controlled layer of change.
- Run hardware, storage, RDMA, MPI/NCCL and application smoke tests.
- Return capacity gradually and observe error/performance baselines.
- Practise restoring scheduler/controller configuration and one representative dataset.
Primary technical references
- Slurm Quick Start Administrator Guide
- Slurm MPI Users Guide
- Kubernetes: Schedule GPUs
- Kubernetes Device Plugins
- NVIDIA GPUDirect RDMA documentation
- Prometheus overview
References are provided for software architecture and implementation planning. Validate the versions, licences, support matrix and regulated-use requirements applicable to the final deployment.