The CUDA and AI Software Stack: From One Hyperion Node to a Cluster

Separate host drivers from application environments

Install and lifecycle-manage the host driver as infrastructure. Put CUDA user-space libraries, Python dependencies and application code inside versioned containers wherever the licensing and workload permit. NVIDIA Container Toolkit exposes GPUs to OCI containers while retaining the host driver boundary. Record image digests, driver versions and GPU firmware in the run metadata.

Do not solve dependency conflicts by upgrading a live production node interactively. Build a new image, run a smoke test and representative benchmark, then promote it through change control.

Framework and library layers

PyTorch, TensorFlow and JAX provide high-level training and inference; cuBLAS, cuDNN and specialised libraries supply accelerated kernels; RAPIDS addresses tabular and distributed data processing; domain frameworks such as MONAI and BioNeMo add validated patterns. Choose the narrowest stack that serves the application and remove unused services from regulated or security-sensitive deployments.

Native CUDA development should establish correctness first, then use Nsight Systems and framework profilers to locate synchronisation, transfer and launch bottlenecks. Optimisation without a measured trace is usually speculation.

Multi-GPU inside one node

Data parallelism replicates a model and divides batches. Tensor or pipeline parallelism divides model state or layers when a model cannot fit one GPU. NCCL supplies topology-aware collectives used by distributed frameworks. Confirm device ordering, CPU affinity and per-process GPU binding; a job that accidentally oversubscribes one device can appear functional while delivering poor performance.

Aggregate VRAM is useful only when the chosen parallel strategy can use it. AxiRelay is naturally suited to independent replicas; AxiForge and AxiForge Max are suited to larger distributed jobs and concurrent experiments.

Scale across nodes only with evidence

PyTorch Distributed and similar frameworks can extend over RDMA-capable networks. Before scaling, benchmark one node, verify data loading and capture communication-to-compute ratio. Then test two nodes with the same global batch and with a growing batch. Configure NCCL/UCX interfaces explicitly and observe retransmits, link utilisation and collective duration.

AxiCrucible is the first Hyperion multi-node AI workgroup; AxiBastion and AxiTitan provide additional GPUs and switched fabric. Scheduling policy and data locality become as important as raw accelerator count.

Inference is a different workload

Inference design balances time-to-first-token or first-result, steady-state throughput, batch delay, context length, model fit and service availability. vLLM targets efficient large-model serving; TensorRT-LLM optimises supported LLM execution; Triton supplies model repositories, concurrent model instances, metrics and dynamic batching. Load tests must use realistic prompt or input distributions rather than a single synthetic request.

Production endpoints need admission control, authentication, request limits, model-version pinning, rollback and telemetry. GPU memory exhaustion must produce a controlled service response rather than destabilising the host.

Reference delivery workflow

1. Pin the host driver and validate the GPU health baseline.
2. Build a signed, immutable application image.
3. Run unit tests, deterministic checks and a representative performance test.
4. Record code commit, image digest, dataset/model version and hardware topology.
5. Deploy through Docker/Podman, Slurm/Apptainer or Kubernetes according to operating model.
6. Monitor latency, throughput, errors, VRAM, power, temperature and queue/service saturation.
7. Promote or roll back as a controlled release.