A dense inference, RAG, CI and multi-service node where several independent GPU workloads must run concurrently.
This brief explains where the baseline fits, the software and operating model it supports, and the evidence Axiotech should use to validate a final configuration. Indicative specifications remain subject to component availability, export compliance and formal quotation.
Baseline architecture
One 4U node; 96 AMD EPYC cores; 768GB ECC DDR5; four NVIDIA RTX PRO 4000 24GB GPUs providing 96GB aggregate VRAM.
The current compute node is based on the Supermicro AS-4125GS-TNRT / CSE-418G2TS 4U rack platform, integrated with matched rails, power, management, networking and rack infrastructure as required.
Best-fit workloads
Model-serving replicas, multiple departmental assistants, medical-imaging services, GPU CI matrices, vision pipelines and other workloads that value concurrency over one large VRAM pool.
Its four independent accelerators suit model replicas, multiple services, concurrent users and pipeline stages better than one oversized GPU.
Recommended software stack
CUDA, TensorRT, TensorRT-LLM, Triton Inference Server, vLLM, Kubernetes or Docker Compose, NVIDIA Container Toolkit, Prometheus and Grafana.
Pin host drivers and infrastructure separately from versioned application containers. Record source, image, dataset/model and hardware allocation with each benchmark or production release.
Deployment pattern
Pin one or more service instances to each GPU through containers, Kubernetes device allocation or an explicit process supervisor. Put authentication, quotas and metrics in front of inference endpoints.
Define monitoring, identity, backup, change control and workload ownership at the same time as compute. Multi-user platforms require resource allocation and quotas; production services require health, overload and rollback behaviour.
Sizing boundary
Aggregate VRAM is not a single memory pool. Models larger than 24GB per GPU require tensor/pipeline parallelism or a larger-memory accelerator such as X1160.
Final sizing should use representative code, data, concurrency and service objectives. Aggregate core, RAM or VRAM figures do not by themselves predict application performance.
Commissioning and acceptance
Load-test all four GPUs concurrently with production-like request sizes, verify per-device isolation, measure p95/p99 latency and demonstrate rolling model replacement without cross-service disruption.
Axiotech should retain the resulting configuration, firmware/driver baseline, environment manifest, benchmark data and recovery procedure as the system acceptance pack.
Primary technical references
References are provided for software architecture and implementation planning. Validate the versions, licences, support matrix and regulated-use requirements applicable to the final deployment.