Reproducible HPC and AI Development: Containers, CI, Profiling and Evidence

Define the run contract

Every production or publishable run should identify the source commit, container or environment digest, parameter file, input dataset version, random seeds, hardware allocation and output location. A manifest written at job start is more reliable than reconstructing provenance later. Keep secrets and personal data out of manifests; store identifiers that point to governed systems.

Containers without losing HPC integration

OCI containers work well for development and services; Apptainer is widely used for batch HPC because it integrates with schedulers and shared filesystems without requiring a long-lived privileged daemon. Keep MPI and accelerator compatibility deliberate: either bind compatible host libraries or build against the deployment standard and test on target hardware.

Use small base images, pinned package versions and software bills of materials. Scan images, sign releases and rebuild regularly rather than carrying an unmaintained environment indefinitely.

CI that tests scientific and accelerated code

Split the pipeline into fast CPU unit tests, accelerator smoke tests, numerical regression tests and scheduled performance tests. Tolerances should reflect floating-point behaviour rather than assuming bitwise identity across every architecture. Preserve a small representative dataset in the test system and keep large or sensitive data behind controlled references.

AxiFoundry can host GPU-enabled runners; AxiGovernor adds controls-software build and simulation; AxiKeystone provides compilation and CPU test density. Queue expensive benchmarks so development feedback remains quick.

Benchmark as a release gate

Define throughput, latency, memory, accuracy and energy or utilisation measures appropriate to the workload. Compare against a named baseline and report variance over multiple runs. Reject changes that silently exceed memory, degrade tail latency or alter scientific outputs outside tolerance. Keep the benchmark harness alongside the source.

Profile the system, not only the kernel

Nsight Systems can expose CPU threads, CUDA calls and GPU activity on a common timeline. Combine it with operating-system, scheduler and storage metrics. A kernel optimisation is irrelevant if the job is waiting for file metadata; a data-loader change may be more valuable than lower kernel duration.

For multi-node runs, add network counters and collective timings. For services, add request traces and queue depth. Profiling should reproduce the real concurrency and input distribution.

Promotion and evidence pack

1. Merge reviewed code to a protected branch.
2. Build, scan and sign the immutable environment.
3. Execute numerical and performance gates on a known Hyperion baseline.
4. Publish artefacts, manifest, test results and benchmark comparison.
5. Deploy with a versioned configuration and rollback target.
6. Review live telemetry before declaring the release complete.