The Infrastructure Layer
In part 1 we mapped the AI stack — application, model, and infrastructure. We established that the infrastructure layer is the platform under the model: the systems that make training and serving possible, and keep them running.
Now we go inside it.
Most engineers who work with AI never see this layer directly. They call an API, deploy to a managed platform, and the infrastructure is abstracted away. That works until it doesn't — until latency spikes, costs spiral, or a serving endpoint falls over under load. When those things happen, the engineers who diagnose and fix it fast are the ones who understand what's actually happening underneath.
This article covers the four core pillars of AI infrastructure: compute, storage, networking, and orchestration — and the platform services built on top of them.
Why AI Infrastructure Is Different
Traditional web infrastructure is optimized for request-response workloads. A server receives a request, queries a database, returns a response. The compute is CPU. The storage is relational. The scale is horizontal but predictable.
AI infrastructure breaks every assumption in that model.
The compute profile is extreme. Inference requires running billions of matrix multiplications at very low latency, millions of times per day. General-purpose CPUs can't do this at the speed production systems need.
The data volumes are massive. Model weights for frontier models are hundreds of gigabytes. Serving them at scale means loading and caching enormous artifacts and delivering fast responses without the weights becoming the bottleneck.
GPU underutilization is expensive. A single H100 GPU costs $30,000 or more. Cloud GPU instances run $3–10 per hour. Every second a GPU sits idle is money wasted.
The performance requirements are unforgiving. Users waiting for a response care about milliseconds. Infrastructure that adds latency at any layer compounds directly into a worse product.
Pillar 1 — Compute
Compute is the foundation. Everything else exists to feed work to compute and get results back out efficiently.
GPUs — The Workhorse of AI
Neural networks are enormous collections of matrix multiplications. A GPU has thousands of small cores designed to run the same mathematical operation on thousands of data points simultaneously — SIMD, Single Instruction Multiple Data. A workload that takes hours on a CPU takes minutes on a GPU.
Current GPU Landscape for Inference
| GPU | VRAM | Memory Bandwidth | Best For | | --- | --- | --- | --- | | NVIDIA H100 | 80GB HBM3 | 3.35 TB/s | Large model inference | | NVIDIA A100 | 80GB HBM2e | 2 TB/s | Production inference | | NVIDIA L40S | 48GB GDDR6 | 864 GB/s | Cost-efficient inference | | AMD MI300X | 192GB HBM3 | 5.3 TB/s | NVIDIA alternative, gaining ground |
VRAM is often the critical constraint. The entire model needs to fit in VRAM during inference. A 70B parameter model in FP16 requires approximately 140GB — two H100s minimum. Engineers who don't understand this end up with out-of-memory errors they can't explain.
CPUs — Still Essential
CPUs handle orchestration logic, API servers, database queries, post-processing of model outputs, and lightweight inference for smaller models.
Custom Accelerators
- AWS Inferentia — Amazon's custom inference chips. Cheaper than H100s, require the AWS Neuron SDK.
- Groq — deterministic latency and extremely high throughput for transformer models.
- Google TPUs — deeply integrated with GCP. Fast for specific workloads, not available outside Google Cloud.
Pillar 2 — Storage
Storage is frequently the bottleneck that gets overlooked. Model weights for frontier models are hundreds of gigabytes. Loading them efficiently and caching them in GPU memory is non-trivial work.
Object storage (S3, GCS, Azure Blob) — the workhorse. Infinite capacity, low cost, accessible from anywhere. Model weights and deployment packages live here.
Model registry — a structured store for managing model versions, metadata, and lineage. MLflow Model Registry and Hugging Face Hub track which version is in production, what metrics it achieved, and who approved the deployment.
The Storage-Compute Gap
One of the most common AI infrastructure failures: engineers provision GPU capacity and watch utilization drop because storage can't deliver model weights fast enough. The bottleneck isn't compute — it's the pipe feeding compute.
For serving: warm caches and inference servers that keep model weights in GPU VRAM across requests rather than reloading each time.
Pillar 3 — Networking
Networking in AI serving infrastructure is about getting requests to models fast and responses back faster.
Load balancers — distribute incoming inference requests across model serving replicas. For AI, you want routing that considers GPU availability and replica health, not just round-robin.
API gateways — the entry point for external traffic. Handle authentication, rate limiting, request validation, and routing before requests reach the model server.
CDN and edge — routing requests to the nearest serving region reduces round-trip time. This matters most for real-time products where every millisecond is felt.
NVLink — when a model spans multiple GPUs, NVIDIA's GPU-to-GPU interconnect within a single server provides 900 GB/s of aggregate bandwidth, allowing large models to be split across multiple GPUs efficiently.
RoCE (RDMA over Converged Ethernet) — high-performance GPU-to-GPU transfers across standard Ethernet for multi-node inference setups.
Pillar 4 — Orchestration
Orchestration is the control plane for all inference workloads running in production.
Kubernetes — The Standard
Kubernetes handles GPU workloads and model serving in production at every major cloud provider.
Pod scheduling — when an inference server needs to scale from 2 replicas to 20, Kubernetes handles placement across available GPU nodes automatically.
Resource management — GPU resource requests enforced by the NVIDIA GPU Operator, which makes GPUs visible to the scheduler.
Health management — if a model serving pod becomes unhealthy (OOM errors, GPU faults, high latency), Kubernetes restarts it and stops routing traffic until it recovers.
Autoscaling — Horizontal Pod Autoscaler scales inference replicas based on CPU, memory, or custom metrics like requests per second. Cluster Autoscaler adds or removes nodes from the underlying cloud.
AI-Specific Kubernetes Tooling
- NVIDIA GPU Operator — automates CUDA driver and runtime installation across every GPU node.
- MIG (Multi-Instance GPU) — partitions a single H100 into up to 7 isolated GPU instances for better utilization.
- Ray Serve — scalable model serving on top of Kubernetes for complex serving pipelines.
Platform Services
Observability
You can't operate what you can't see.
Infrastructure metrics — GPU utilization, GPU memory usage, inference latency, request throughput, error rates. Prometheus collects, Grafana displays.
Model metrics — token usage per request, generation latency per token, output quality scores. These tie directly to cost and user experience.
Distributed tracing — following a single request through the entire stack from API gateway through model server to GPU and back. OpenTelemetry is the standard.
Security and Access Control
Identity and access management — who can access model endpoints and who can deploy to production. IAM policies, service accounts, and RBAC in Kubernetes.
Secrets management — API keys and credentials should never be stored in code or container images. Vault, AWS Secrets Manager, and Kubernetes Secrets handle injection at runtime.
Model security — model weights are valuable intellectual property. Access controls and encryption at rest protect them.
Deployment Environments
Production deployments span multiple environments, often simultaneously.
Cloud (GCP, AWS, Azure) — the default for most teams. Elastic scaling, managed Kubernetes, latest GPU hardware without capital expenditure. Trade-off: cost at scale.
On-premises — for compliance requirements or predictable workloads. More control and lower per-unit cost at scale, but significant capex and operational complexity.
Edge / on-device — smaller models on local hardware where low latency, privacy, and offline operation matter. Requires model compression to fit capable models on constrained hardware.
Decentralized compute — Akash and io.net allow anyone to contribute GPU capacity to a shared pool. Lower cost than hyperscalers, less mature tooling.
In 2026, most production AI systems are hybrid: cloud for elastic inference, on-premises for sensitive workloads, edge for latency-critical inference, and decentralized compute for cost optimization.
Summary
| Pillar | What it does | Key technologies | | --- | --- | --- | | Compute | Runs inference — GPU-accelerated, low latency | H100/A100/L40S GPUs, Groq, Inferentia | | Storage | Stores and serves model weights and artifacts | S3, GCS, Model Registry | | Networking | Routes requests to models and responses back | Load balancers, API gateways, NVLink | | Orchestration | Manages serving workloads across the cluster | Kubernetes, GPU Operator, Ray Serve |
The infrastructure layer is where the bottlenecks hide. It's where latency is born, where costs accumulate, and where reliability is either built in or absent. Getting it right is what separates AI products that scale from AI products that struggle.