The Model Layer
In part 1 we mapped the AI stack — application, model, and infrastructure. In part 2 we went inside the application layer: agents, orchestration, tools, and runtime environments.
Now we go inside the model.
Most engineers who build with AI interact with the model layer through an API. Send a prompt, receive a response. The model is a black box. That works until it doesn't — until you need to understand why the model is behaving a certain way, choose between competing models for a production task, decide whether to fine-tune or RAG, or estimate what infrastructure you need to serve it.
This article opens the black box. What models actually are, how they come to exist, how they run, and what the full landscape looks like in 2026.
What a Model Actually Is
A neural network is a function. It takes an input, runs it through a series of mathematical transformations, and produces an output.
input → [layer 1] → [layer 2] → ... → [layer N] → output
The transformations at each layer are controlled by weights — numbers. Billions of them. The weights are what the model learned. Everything the model knows about language, reasoning, code, and the world is encoded in those numbers. Change the weights and you change what the model produces.
Parameters is the word you'll see in model descriptions — "a 70B parameter model." Parameters and weights mean the same thing in this context: the total count of learnable numbers in the model. More parameters, more capacity to represent complex patterns, more compute required to run it.
The Transformer Architecture
Modern LLMs are all built on the Transformer architecture, introduced by Google in 2017. The key mechanism is attention — a way for the model to decide which parts of the input are most relevant when producing each output token.
You don't need to understand the math. Two things matter practically:
Context window — the transformer can only attend to a fixed number of tokens at once. This is the context window: 8K, 128K, 200K tokens depending on the model. Longer context = the model can reason over more information at once, but also = more compute per token and more memory required.
Scale — larger models (more parameters) are generally more capable. The relationship isn't perfectly linear, but it's consistent enough that model size is a reliable proxy for capability across most tasks. A 70B model outperforms a 7B model on most benchmarks — at roughly 10x the infrastructure cost.
┌─────────────────────────────────────────────────────┐
│ TRANSFORMER BLOCK │
│ │
│ Input tokens │
│ ↓ │
│ Token embeddings (text → vectors) │
│ ↓ │
│ Attention layers (which tokens matter most?) │
│ ↓ │
│ Feed-forward layers (transform the representation) │
│ ↓ │
│ Repeated N times (N = number of layers) │
│ ↓ │
│ Output: probability distribution over next token │
└─────────────────────────────────────────────────────┘
Training
Training is how models come to exist. It's the process of adjusting weights until the model produces useful outputs.
Data
Training starts with data. For a large language model, that means text — enormous quantities of it. Common Crawl (a snapshot of much of the internet), books, code repositories, Wikipedia, scientific papers, and curated high-quality sources.
Data quality matters more than data quantity past a certain scale. Models trained on carefully filtered, deduplicated, high-quality text outperform models trained on more raw data. Data curation — filtering spam, removing duplicates, balancing domains, deduplicating near-identical passages — is a significant engineering problem in itself.
Tokenization — text doesn't go into the model as characters or words. It gets split into tokens: subword units that balance vocabulary size against sequence length. "Unbelievable" might tokenize to ["un", "believ", "able"]. A typical English word is 1–2 tokens. Code is often more. The tokenizer is trained separately, then fixed before model training starts.
Pretraining
The core of training is pretraining — running the model through massive amounts of text and teaching it to predict the next token.
Input: "The transformer architecture was introduced"
Target: "in" ← model predicts this token
The model makes a prediction. The prediction is compared to the actual next token. The difference (the loss) flows backward through the network (backpropagation), adjusting weights slightly to make the correct prediction more likely. Repeat this billions of times across trillions of tokens.
That's it. The entire intelligence of a frontier LLM emerges from next-token prediction at scale. The model learns grammar, facts, reasoning patterns, and code structure because predicting the next token well requires understanding all of them.
Why training is expensive:
- Frontier models train on 10–15 trillion tokens
- Training requires thousands of GPUs running in parallel for weeks to months
- GPT-4 was estimated to cost over $100M to train
- Llama 3.1 405B took 30,720 H100 GPUs across the full training run
- A single training run failure or instability can cost millions of dollars
This is why almost no one trains from scratch. You start from a pretrained model.
Alignment (RLHF / RLAIF)
A pretrained model is a next-token predictor. It's not necessarily helpful, safe, or good at following instructions. Alignment is the process of shaping behavior after pretraining.
RLHF (Reinforcement Learning from Human Feedback) — human raters compare model outputs and rank them. A reward model is trained to predict human preferences. The LLM is then fine-tuned using reinforcement learning to maximize the reward signal. This is how models learn to be helpful and avoid harmful outputs.
RLAIF (Reinforcement Learning from AI Feedback) — uses another AI model as the rater instead of humans. Scales better, cheaper, but depends on the quality of the AI rater. Constitutional AI (Anthropic's method) is the most developed RLAIF approach.
DPO (Direct Preference Optimization) — a simpler alternative to RLHF that achieves similar results without a separate reward model. Increasingly common in 2026 for production alignment.
Inference
Training happens once. Inference happens millions of times per day. It's where your production costs live.
Autoregressive Generation
LLMs generate text one token at a time. Each token is produced by running a full forward pass through the model, then the new token is appended to the sequence and the process repeats.
Prompt: "The sky is"
Step 1: model sees "The sky is" → generates "blue"
Step 2: model sees "The sky is blue" → generates "."
Step 3: model sees "The sky is blue." → generates [EOS]
This is autoregressive generation. Each token depends on all previous tokens. You can't parallelize token generation — step 2 can't start until step 1 completes. This is the fundamental constraint that makes LLM inference latency proportional to output length.
KV Cache
Every forward pass requires computing attention over all previous tokens. Without optimization, this is O(n²) compute as sequences get longer.
The KV cache solves this by storing the key and value matrices for all previously processed tokens. On each new generation step, only the new token needs to be processed — the cached computations for prior tokens are reused.
KV cache is why inference gets progressively cheaper per token as the sequence grows (up to the point where cache memory becomes the bottleneck). It's also why long-context inference consumes significantly more GPU memory than short-context inference.
Batching
Running one request at a time leaves most of the GPU idle — the GPU is waiting for the memory-bound KV cache operation rather than doing compute. Batching runs multiple requests through the model simultaneously.
Continuous batching (also called in-flight batching) — rather than grouping requests into fixed batches, new requests join the batch as slots open. This keeps GPU utilization high even when requests have different lengths. It's the standard approach in production inference servers.
Quantization
Model weights are stored as floating-point numbers. The default is FP32 (32 bits per weight). A 70B model in FP32 requires 280GB — four H100 GPUs just to hold the weights.
Quantization reduces precision to reduce memory and increase speed:
| Format | Bits per weight | 70B model size | Quality impact | | --- | --- | --- | --- | | FP32 | 32 | 280 GB | Full precision (reference) | | FP16 / BF16 | 16 | 140 GB | Negligible loss | | INT8 | 8 | 70 GB | Slight degradation on complex reasoning | | INT4 | 4 | 35 GB | Noticeable on nuanced tasks, acceptable for many | | GGUF Q4_K_M | ~4.5 | ~40 GB | Common format for local inference |
The tradeoff: lower precision = smaller model = faster inference = lower hardware requirement = some quality loss. For most production use cases, FP16 or INT8 quantization is the right choice. INT4 is common for edge deployment or cost-sensitive batch workloads.
Inference Engines
Running a model efficiently in production requires more than loading weights and calling forward. Inference engines handle batching, KV cache management, quantization, and GPU memory allocation:
- vLLM — the most widely deployed open-source inference engine. PagedAttention (treats KV cache like virtual memory pages) dramatically improves throughput. The default choice for self-hosted serving.
- TensorRT-LLM — NVIDIA's inference library. Compiles models to optimized CUDA kernels. Higher ceiling for throughput than vLLM on NVIDIA hardware, harder to configure.
- Ollama — developer-focused local inference. Simple to run, handles model downloads and quantization automatically. Not for production at scale, excellent for development.
- SGLang — efficient serving for complex generation patterns (multi-turn, structured output, long context). Growing adoption in 2026.
Fine-tuning
A pretrained model is general. Fine-tuning makes it specialized.
When to Fine-tune
Three approaches for adapting a model to a specific task:
| Approach | When to use | Cost | Tradeoffs | | --- | --- | --- | --- | | Prompt engineering | General tasks, behavior already in the model | Near-zero | Limited control, longer prompts = higher cost | | RAG | Factual knowledge retrieval, large document sets | Low–medium | Retrieval quality bottleneck, still needs a capable base model | | Fine-tuning | Specific style/format/behavior, task performance, cost reduction | Medium–high | Requires data, training infrastructure, evaluation |
Fine-tune when the behavior you need isn't reliably achievable through prompting alone, when you have hundreds to thousands of labeled examples, or when you need to run a smaller model with the capability of a larger one to manage cost.
Full Fine-tuning vs PEFT
Full fine-tuning — update all model weights during training. Maximum adaptation, highest quality ceiling, requires storing a full copy of the model for each fine-tuned variant. A full fine-tune of Llama 3.3 70B requires a multi-GPU cluster and days of training time.
PEFT (Parameter-Efficient Fine-tuning) — update only a small subset of parameters. The base model stays frozen; only lightweight adapter layers are trained.
LoRA (Low-Rank Adaptation) — the dominant PEFT method. Instead of updating full weight matrices, LoRA trains small low-rank matrices that are added to the original weights. Typical LoRA adapters are <1% of the model's parameter count.
Original weight: W (70B parameters, frozen)
LoRA adapter: A × B (millions of parameters, trained)
Effective weight during inference: W + A × B
QLoRA — LoRA on a quantized base model. Quantize the base model to INT4 to reduce memory, train LoRA adapters in FP16. A 70B model fine-tune that normally requires 8×A100s fits on 2×A100s with QLoRA.
Real costs:
- LoRA fine-tune of Llama 3.1 8B: single A100, 2–4 hours, ~$10–30 on cloud compute
- QLoRA fine-tune of Llama 3.3 70B: 2–4×A100s, 1–2 days, ~$200–800
- Full fine-tune of a 70B model: 8–32×A100s, 3–7 days, $2,000–20,000+
Alignment Fine-tuning
Fine-tuning for behavior and safety (not just task performance):
SFT (Supervised Fine-tuning) — train on (instruction, ideal response) pairs. Teaches the model to follow a specific format, maintain a persona, or produce outputs in a particular style.
DPO (Direct Preference Optimization) — train on (instruction, preferred response, rejected response) triplets. Directly optimizes the model to prefer the better output without a separate reward model. Simpler than RLHF, increasingly the standard in 2026.
Model Types in 2026
The model layer isn't one thing. Different tasks require different model architectures.
Large Language Models (LLMs)
Text understanding and generation. The general-purpose layer. Everything from answering questions to writing code to summarizing documents.
| Model | Parameters | Notes | | --- | --- | --- | | Claude Opus / Sonnet | Undisclosed | Anthropic flagship — strong reasoning, long context | | GPT-4o | Undisclosed | OpenAI flagship — multimodal, fast | | Llama 3.3 70B | 70B | Meta open-source — competitive with frontier on many tasks | | Mistral Large | Undisclosed | Strong European model, aggressive pricing | | Qwen 2.5 72B | 72B | Alibaba open-source — excellent on coding and multilingual | | DeepSeek V3 | 685B (MoE) | Strong open-source, Chinese lab, cost-efficient |
Reasoning Models
Optimized for multi-step chain-of-thought reasoning. Spend more compute thinking before answering. Significantly better on math, logic, and complex planning — at higher latency and cost.
| Model | Notes | | --- | --- | | o3 | OpenAI — best-in-class reasoning, expensive | | o4-mini | OpenAI — fast reasoning at lower cost | | DeepSeek-R1 | Open-source reasoning model, competitive with o3 on benchmarks | | Claude with extended thinking | Anthropic's approach — thinking budget is configurable |
Use reasoning models when the task requires multiple inferential steps and correctness matters more than speed. Don't use them for simple tasks — you'll pay for thinking time you don't need.
Vision Language Models (VLMs)
Image and text together. The model can see.
- GPT-4o — native multimodal, processes text and images in a unified model
- Gemini 1.5 / 2.0 — Google's multimodal models, strong on long-context vision tasks
- LLaVA, Pixtral, InternVL — open-source VLMs, rapidly catching up to frontier
Use cases: document understanding, image captioning, chart analysis, screenshot-to-code, visual QA.
Embedding Models
Convert text into dense vectors. These aren't generation models — they don't produce text. They produce representations that capture semantic meaning, used for similarity search and retrieval.
| Model | Dimensions | Notes | | --- | --- | --- | | text-embedding-3-large | 3072 | OpenAI — strong general embedding | | Cohere embed-v3 | 1024 | Good multilingual, supports input types | | nomic-embed-text | 768 | Open-source, local deployment | | bge-m3 | 1024 | Strong open-source, multilingual |
Every RAG pipeline depends on embedding quality. A better embedding model improves retrieval directly, which improves generation quality indirectly. Embedding model choice is underrated in most RAG implementations.
Audio and Multimodal
- Whisper — OpenAI's speech-to-text model. Open-source, highly accurate, standard for transcription
- GPT-4o native audio — real-time audio understanding and generation, low latency for voice products
- Gemini 1.5 — native audio, video, and text in a single model
Open-Source vs Frontier
The gap is closing. In 2026, the choice between open-source and frontier API is a real engineering decision, not a default.
Capability in 2026
| Task | Frontier API | Best Open-Source | | --- | --- | --- | | Complex reasoning | o3, Claude Opus | DeepSeek-R1 (competitive) | | General instruction following | GPT-4o, Claude Sonnet | Llama 3.3 70B, Qwen 2.5 72B | | Coding | Claude Sonnet, GPT-4o | Qwen 2.5 Coder 72B, DeepSeek V3 | | Long context | Gemini 1.5, Claude | Llama 3.1 (128K), Qwen (128K) | | Multilingual | GPT-4o, Gemini | Qwen 2.5, Mistral | | Local / edge | N/A | Llama 3.2 3B, Phi-3.5, Gemma 2 |
Llama 3.3 70B passes GPT-4 (original) on MMLU and HumanEval. Qwen 2.5 Coder 72B matches GPT-4o on coding benchmarks. Frontier still leads on complex multi-step reasoning and nuanced instruction following — but the gap on most production tasks is narrow.
Cost Analysis
Frontier API pricing (approximate, 2026):
- Claude Sonnet: ~$3 / million input tokens, ~$15 / million output
- GPT-4o: ~$2.50 / million input, ~$10 / million output
- GPT-4o-mini: ~$0.15 / million input, ~$0.60 / million output
Self-hosted open-source (rough compute cost for vLLM on A100s):
- Llama 3.1 8B: ~$0.05–0.10 / million tokens all-in
- Llama 3.3 70B: ~$0.30–0.60 / million tokens at scale
- DeepSeek V3 (MoE): highly efficient, ~$0.14 API / million input tokens via their API
At low volume, frontier APIs win on cost because you pay only for usage. At scale (millions of tokens per day), self-hosted open-source is significantly cheaper — often 5–10x.
When to Use Each
Frontier API:
- Prototyping and low-volume applications
- Tasks where quality at the frontier matters (complex reasoning, nuanced generation)
- When you don't have the infrastructure team to run your own models
- Regulated industries where you need a vendor with compliance certifications
Self-hosted open-source:
- High-volume applications where cost at scale matters
- Data privacy requirements (no data leaving your infrastructure)
- Fine-tuning for specific domain behavior
- Latency-critical applications that need GPU co-location with your application
Hybrid: route simple tasks to a cheap open-source model, complex tasks to a frontier API. Model routing with LiteLLM or OpenRouter handles this at the application layer.
How Model Quality Affects Everything Else
The model layer is the center of gravity. Everything in the stack exists to support it or is constrained by it.
Model quality determines product quality. A brilliant application layer can't compensate for a weak model. If the model hallucinates, the product hallucinates. If the model can't follow multi-step instructions reliably, the agent fails. The application layer multiplies model quality; it doesn't replace it.
Model choice drives infrastructure requirements. A 70B model requires different hardware than an 8B model — more VRAM, more nodes for multi-GPU serving, higher memory bandwidth. The infrastructure layer is provisioned around the model. Change the model and you change the infrastructure requirements.
Model size → VRAM requirement → GPU selection → cluster topology
Model latency profile → serving strategy → autoscaling parameters
Model context window → KV cache size → memory per request
Model capabilities bound what the application can build. A model that can't reliably use tools won't power a capable agent. A model without long context can't reason over large codebases. A model without strong instruction following can't be aligned to specific personas or constraints. Application layer engineering is constrained by what the model layer can do.
Inference cost drives unit economics. The model is the most expensive component per request in most AI applications. Application layer latency budgets, feature design, and pricing are all downstream of inference cost. A cheaper, faster model at equivalent quality changes what products are viable.
Summary
| Concept | What it is | Why it matters | | --- | --- | --- | | Weights / parameters | Learned numbers encoding model knowledge | What the model "knows" — change them, change behavior | | Transformer / attention | Architecture enabling context-aware token prediction | Why context windows matter; why scale improves capability | | Pretraining | Next-token prediction on trillions of tokens | Where model intelligence comes from | | RLHF / DPO | Alignment via human or AI preference data | Makes models helpful, not just coherent | | Inference | Autoregressive token-by-token generation | The production bottleneck — latency and cost live here | | KV cache | Cached attention keys/values for prior tokens | Why long inference isn't as slow as it could be | | Quantization | Reducing weight precision (FP32→FP16→INT4) | Smaller model, faster inference, manageable quality tradeoff | | vLLM / TensorRT-LLM | Production inference engines | How you serve models efficiently at scale | | Fine-tuning | Updating weights for specific task/domain | Better task performance without training from scratch | | LoRA / QLoRA | Parameter-efficient fine-tuning | Fine-tune large models with modest hardware | | Embedding models | Text → dense vectors | Foundation of every RAG and semantic search system | | Open-source models | Llama, Qwen, Mistral, DeepSeek | Frontier-competitive quality at significantly lower cost at scale |
The model layer is the only layer that can't be engineered around. Infrastructure exists to run it. Applications exist to use it. Understanding what models actually are — how they're built, how they run, what they can and can't do — is the foundation of effective AI engineering.
Next in the series: The Infrastructure Layer — compute, storage, networking, and orchestration. The platform that makes the model layer possible.