Designing Secure In-House LLM Platforms with Hyperion

Three service tiers

X1440 AxiRelay favours four replicas or independent models that each fit 24GB. X1160 AxiAnvil favours one larger model in 96GB with simple deployment. X4460 AxiTitan supports organisation-scale multi-team serving, distributed models, evaluation and fine-tuning across sixteen 96GB GPUs.

Choose on measured context, quantisation, concurrency and latency. Parameter count alone is not a sizing method.

Serving stack

vLLM supplies high-throughput LLM serving and documented tensor/pipeline parallel options. TensorRT-LLM can optimise supported models. Triton provides a managed model repository, metrics, multiple instances and dynamic batching. Put an authenticated gateway in front for quotas, request size, timeouts and audit policy.

Permission-aware retrieval

Ingest only approved sources. Preserve owner, effective date, classification and access group at chunk level. Filter retrieval before content reaches the model; post-filtering an answer is too late. Return citations and timestamps, and provide a clear “insufficient evidence” path. Re-index on revocation or document change.

Security and model governance

Segment management, model-serving and data networks. Sign container images, pin model checksums and restrict egress. Treat prompts, retrieved passages and outputs as potentially sensitive. Protect against prompt injection by separating instructions from untrusted content, limiting tools and validating actions outside the model. Guardrail frameworks can help express controls but do not replace application authorisation.

Evaluation and capacity

Build task-specific evaluations for factuality, citation correctness, refusal, leakage, harmful actions and latency. Load-test prompt length and concurrency distributions from the intended service. Watch time-to-first-token, tokens per second, queue time, cache hit, GPU memory and error rate. Capacity plans should include peak demand and maintenance headroom.

Hybrid cloud without uncontrolled leakage

A gateway may route explicitly approved low-sensitivity tasks to external services while keeping restricted work local. Classification and user consent must occur before transmission. Log the policy decision and provider/model used without retaining confidential prompt text unnecessarily. Provide a fail-closed option for protected workflows.