This pattern adds GPUs to a conventional CI estate without turning the server into a shared interactive workstation. Jobs receive explicit devices, immutable environments and evidence that ties a result to hardware and software versions.
Runner roles
AxiKeystone hosts parallel builds and CPU test matrices. AxiFoundry provides one development GPU for serial accelerator CI. AxiRelay allows four isolated GPU jobs or compatibility lanes. Use ephemeral runner identities and separate protected-release runners from untrusted contributions.
Pipeline layers
- Format, lint and CPU unit tests.
- Build OCI/Apptainer image and generate SBOM.
- GPU discovery and minimal kernel/framework smoke test.
- Numerical regression on a fixed dataset.
- Integration and multi-process tests.
- Scheduled representative performance benchmark.
- Sign/promote artefacts only after gates pass.
Device isolation
Allocate GPUs by runner or scheduler, set visible devices and verify no foreign process is present. Record GPU, driver and CUDA runtime. Reset failed jobs cleanly; persistent GPU state can contaminate later results. Never expose host device management or privileged containers to arbitrary builds.
Performance gate
Profile a realistic workload with Nsight Systems, not only a microkernel. Compare median and variance across repeated runs; collect host/device transfer, GPU utilisation, memory high-water mark and wall time. Alert on statistically meaningful regressions and retain the trace for investigation.
Security
Pin and scan images, protect package mirrors, remove long-lived credentials and use signed release artefacts. Source code supplied to a private assistant must follow repository permissions. Generated code passes the same review and tests as human code.
Primary technical references
- NVIDIA Container Toolkit overview
- NVIDIA CUDA C++ Best Practices Guide
- NVIDIA Nsight Systems User Guide
- Kubernetes Device Plugins
References are provided for software architecture and implementation planning. Validate the versions, licences, support matrix and regulated-use requirements applicable to the final deployment.