The Application Layer

In part 1 we mapped the AI stack at the highest level: application, model, and infrastructure. The application layer is where products live — it's what users and other systems interact with directly.

Now we go inside it.

The application layer in 2026 is dominated by agents. Not just chatbots — real agents that reason, use tools, maintain memory, execute code, and coordinate with other agents to complete complex tasks. An agent isn't just a prompt with a loop. It's a layered system — each layer doing a specific job, each depending on the one beneath it. This article maps that stack from top to bottom.

Why Agents Need Their Own Stack

Traditional web infrastructure was built for stateless request-response. A request comes in, a server processes it, a response goes out. State lives in a database. The server forgets everything between requests.

Agents break this model entirely.

Durable state across multi-step execution — an agent working on a complex task might take 50 steps over 20 minutes. That state needs to persist reliably across every step, survive failures, and be resumable if interrupted.

Isolated execution environments — agents execute code, run shell commands, and interact with external systems autonomously. Running untrusted AI-generated code in a shared environment is a security disaster. Every execution needs isolation.

Persistent memory across sessions — a truly capable agent remembers what it did last week, what the user prefers, what context is relevant. That's a different problem from in-context retrieval.

Tool access with governance — agents call APIs, write to databases, send emails, and execute transactions. That access needs to be controlled, audited, and rate-limited.

Coordination primitives — multi-agent systems need agents to hand off tasks, share context, and coordinate without coupling directly to each other's implementations.

Behavioral observability — you can't debug an agent with standard application metrics. You need to understand what the agent decided, why it made each tool call, and where it went wrong.

The Problem With How Most People Think About Agents

Most developers think about agents like this:

User input → LLM → Output

That's not an agent. That's a chatbot.

A real agent looks more like this:

User input → reasoning → tool calls → memory → more reasoning → output

And a production agent — one that runs reliably, at scale, for real users or systems — looks like this:

User input
    ↓
Orchestration layer
    ↓
Execution layer
    ↓
Runtime environment
    ↓
Agent (model + tools + memory + state + harness)
    ↓
Infrastructure
    ↓
Cloud / compute

Each of those layers is doing real work. Let's go through them one by one.

Layer 1 — The Model Layer

At the core of every agent is a model. This is the reasoning engine — the LLM that takes input, thinks, and decides what to do next.

The model doesn't act on its own. It generates text. Specifically, it generates one of two things:

A response — a direct answer to return
A tool call — an instruction to call an external function

The model doesn't execute tool calls itself. It decides that a tool call should happen and outputs the instruction. Something else — the harness — picks that up and runs it.

What the model layer includes

The LLM (GPT-4o, Claude, Llama, Mistral)
The context window — everything the model can see at once
The system prompt — the agent's identity and instructions
Token management — staying within context limits

The key constraint: The model is stateless. It has no memory between calls. Every call starts fresh. Memory and state are handled by layers above it.

Production agents rarely use one model for everything. Different steps have different requirements — planning needs strong reasoning, tool call formatting needs speed, code generation needs coding ability. Model routing directs each step to the most appropriate and cost-efficient model. LiteLLM and OpenRouter handle routing, fallback, cost tracking, and policy enforcement across providers.

Step                    Model
Task planning           o3, Claude (strong reasoning)
Tool call formatting    GPT-4o-mini, Haiku (fast, cheap)
Code generation         Claude 3.5 (coding ability)
Document summarization  Llama 3 (local, cost-efficient)
Final response          Claude, GPT-4o (quality)

Layer 2 — The Agent Layer

The agent isn't just the model. The agent is the full unit of intelligence — everything needed for the model to reason, act, and maintain continuity across a task.

┌─────────────────────────────────────┐
│              AGENT                  │
│                                     │
│  Model ──── reasons and decides     │
│  Tools ──── acts on the world       │
│  Memory ─── recalls past context    │
│  State ──── tracks task progress    │
│  Harness ── connects to environment │
└─────────────────────────────────────┘

Tools

Tools are functions the agent can call. They extend what the agent can do beyond generating text.

Examples:

search(query) — search the web
execute_code(code) — run a Python script
call_api(endpoint, payload) — hit an external service
read_file(path) — read from a file system
send_payment(amount, recipient) — execute a transaction

The model decides which tool to call and with what arguments. The harness executes it. The result comes back into the model's context.

MCP changed the tools layer. The Model Context Protocol, released by Anthropic in late 2024, standardized how agents connect to tools. Before MCP, every framework had its own integration format. Now a tool implemented as an MCP server works with any MCP-compatible agent. Two other protocols matter in 2026: A2A (Agent-to-Agent, Google's standard for agent-to-agent handoffs) and x402 (HTTP 402-based micropayments — agents pay for tool calls onchain without human involvement).

In production, the tools layer needs governance: agents should never see raw credentials (injected at the network boundary via a proxy), access control enforcing which agents can call which tools, rate limiting to prevent a single agent exhausting an API quota in minutes, and audit logging of every tool call with agent identity, inputs, outputs, and timestamp.

Memory

The model forgets everything between calls. Memory systems solve this by persisting context outside the model.

Four tiers, each serving a different purpose:

In-context memory — the current conversation history, passed in every call. Fast but limited by context window size (128K–200K tokens for frontier models).
Episodic memory — what happened in past sessions. User preferences, prior decisions, conversation history. Persisted in Postgres or Redis with vector embeddings for retrieval. Mem0 is the leading purpose-built solution.
Semantic memory — structured knowledge the agent retrieves when relevant. This is where RAG lives — chunked, embedded, indexed in a vector database. Qdrant, Weaviate, Pinecone, and Chroma are the common stores. Retrieval quality is determined by chunking strategy, embedding model, and reranking — not just the database itself.
Procedural memory — how to do things. Successful tool call sequences, effective prompting patterns, learned workflows. Still largely research territory in 2026.

State

State tracks where the agent is in a task. Not what it remembers — what it's currently doing.

Examples:

Is this agent idle or mid-task?
What step of a multi-step workflow is it on?
Has a circuit breaker been triggered?

State is a machine concern, not a model concern. The model reads state as context. The state machine manages transitions.

Harness

The harness is the adapter that connects the agent to its execution environment. It's the interface between the agent's logic and the infrastructure running it.

The harness does four things:

Receives a task from the execution layer
Starts the agent — feeds it the task, context, and available tools
Intercepts tool calls the model generates and executes them
Returns results and progress back to the execution layer

Without a harness, the agent logic isn't runnable. The harness is what makes a specific agent (Claude Code, a custom LangGraph agent, your own agent loop) pluggable into any execution platform.

Think of it as a power socket standard. Any agent that conforms to the harness interface can be plugged in and run. The execution platform doesn't need to know what the agent is — it just talks to the harness.

Layer 3 — The Orchestration Layer

Orchestration is what coordinates multiple agents, or multiple steps of a single agent, toward a goal.

A single agent handles one task. Orchestration handles workflows — sequences, branches, parallel execution, and handoffs between agents.

┌─────────────────────────────────────┐
│        ORCHESTRATION LAYER          │
│                                     │
│  task routing                       │
│  agent-to-agent communication       │
│  workflow sequencing                │
│  state coordination                 │
│  error handling and retries         │
└─────────────────────────────────────┘

Common orchestration frameworks

LangGraph — graph-based agent workflows with explicit state management
CrewAI — role-based multi-agent coordination
AutoGen — conversational multi-agent patterns
Inngest — durable background workflows with step functions, sleep/wait, and event-driven execution
Temporal — enterprise-grade durable workflows that survive service restarts and handle long-running tasks
Custom loops — what production teams often end up writing once frameworks become a bottleneck

The trend in 2026 is toward custom orchestration loops for production systems. Frameworks are excellent for prototyping. Production teams regularly hit their limits and rewrite.

Layer 4 — The Execution Layer

The execution layer is where the agent actually gets run. It's the infrastructure-aware layer that takes an agent task from orchestration and makes it happen on real compute.

┌─────────────────────────────────────┐
│          EXECUTION LAYER            │
│                                     │
│  spawn runtime environment          │
│  inject credentials and tools       │
│  manage execution lifecycle         │
│  persist execution state            │
│  stream results back                │
│  tear down on completion            │
└─────────────────────────────────────┘

The execution layer is agent-agnostic. It doesn't know or care what kind of agent is running — it provides a sandboxed environment for the harness to operate in, manages the lifecycle, and persists everything that happened.

Spawning — when a task comes in, a new isolated environment is created. In production this is typically a container or microVM — an isolated unit with a shell, workspace, and the agent's dependencies.

Credential injection — agents need API keys, database credentials, and service tokens. The execution layer injects these at runtime without exposing raw secrets to the agent environment. The agent sees a placeholder; the real credential is swapped in at the network boundary.

Lifecycle management — start, run, monitor, and tear down executions. Handle timeouts, retries, and failures.

Persistence — every execution is logged. Messages, tool calls, events, intermediate results, and the final output are stored so they can be retrieved, audited, or replayed. If the client disconnects, the result isn't lost.

Streaming — results don't wait until the agent finishes. The execution layer streams progress back in real time.

Layer 5 — The Runtime Environment

The runtime environment is the actual sandbox where the agent runs. It's the physical instantiation of the execution layer's spawn operation.

┌────────────────────────────────────────┐
│         RUNTIME ENVIRONMENT            │
│         (Isolated Sandbox)             │
│                                        │
│  shell and workspace                   │
│  agent dependencies and libraries      │
│  tool endpoints accessible             │
│  network policy: default deny          │
│  outbound traffic through proxy only   │
│  resource limits enforced              │
└────────────────────────────────────────┘

Each execution gets its own isolated runtime. Isolation means one agent's execution can't affect another's — separate filesystem, separate network namespace, separate resource allocation.

Why Isolation Is Non-Negotiable

Agents execute LLM-generated code. LLM-generated code is untrusted by definition — the model may hallucinate dangerous operations, misunderstand context, or be manipulated by adversarial inputs. Running that code in a shared environment risks data exfiltration, resource exhaustion, and lateral movement.

In 2026, shared-kernel container isolation (standard Docker) is no longer considered sufficient. The consensus has moved to hardware-enforced isolation.

Firecracker microVMs — AWS's open-source microVM technology. Boots in ~125ms with a minimal kernel. Hardware-enforced isolation — each sandbox runs in its own virtual machine. The gold standard for untrusted code execution. Used by AWS Lambda, Fly.io Sprites, and E2B.

gVisor — Google's container sandbox that runs a full emulated Linux kernel in userspace. Strong isolation without full VM overhead. Used by Modal and Google's own agent infrastructure.

Kata Containers — combines container tooling with VM-level isolation using hardware virtualization. Open-source, Kubernetes-compatible.

Memory snapshotting addresses cold start. Checkpointing the sandbox state after initialization means subsequent starts restore from snapshot rather than initializing from scratch — cutting startup from seconds to milliseconds.

| Platform | Isolation | Best For | | --- | --- | --- | | E2B | Firecracker microVM | Developer-friendly, fast SDK integration | | Modal | gVisor | ML-heavy workloads, GPU access | | Fly.io Sprites | Firecracker | Stateful long-running agent sessions | | Daytona | Firecracker | Dev environments, repo-based agents | | Northflank | Kata Containers | Full-stack with BYOC cloud support | | DIY on K8s | gVisor or Kata | Maximum control, own the infrastructure |

Layer 6 — Agent Infrastructure

Agent infrastructure is the platform that runtime environments run on — the Kubernetes cluster, networking, messaging, observability, and cloud resources that make everything above it possible.

┌────────────────────────────────────────┐
│          AGENT INFRASTRUCTURE          │
│                                        │
│  Kubernetes cluster                    │
│  VPC, subnets, NAT, firewall rules     │
│  Inter-agent messaging (NATS, Redis)   │
│  Ingress and load balancing            │
│  Prometheus + Grafana observability    │
│  Terraform-provisioned cloud resources │
└────────────────────────────────────────┘

This layer knows nothing about agents. It provides reliable, scalable, observable compute for whatever workloads run on it.

Messaging — agents in a multi-agent system need to communicate. A message bus like NATS JetStream provides low-latency, persistent pub/sub between agent processes. Each agent subscribes to its channel. The orchestrator publishes tasks. Results flow back through the same bus.

Observability — Prometheus scrapes metrics from every pod. Grafana surfaces them as dashboards. You want to see: execution count, latency per agent, tool call volume, error rates, and resource consumption per execution.

Autoscaling — agent workloads are bursty. Kubernetes Horizontal Pod Autoscaler scales execution capacity up and down based on demand, so you're not over-provisioned at idle and not bottlenecked at peak.

Resource limits — agent executions can consume unpredictable amounts of CPU and memory, especially when running code or processing large contexts. Resource requests and limits on each pod prevent any single execution from consuming the cluster.

The Full Stack Together

┌──────────────────────────────────────────────┐
│              USER / CLIENT                   │
│         CLI · API · Web Dashboard            │
└──────────────────────┬───────────────────────┘
                       │ submits task
┌──────────────────────▼───────────────────────┐
│           ORCHESTRATION LAYER                │
│      LangGraph · CrewAI · Custom Loop        │
│                                              │
│  coordinates agents                          │
│  manages workflow and task routing           │
└──────────────────────┬───────────────────────┘
                       │ triggers execution
┌──────────────────────▼───────────────────────┐
│             EXECUTION LAYER                  │
│                                              │
│  spawns runtime environment                  │
│  manages lifecycle and persistence           │
│  streams results                             │
└──────────────────────┬───────────────────────┘
                       │ spins up
┌──────────────────────▼───────────────────────┐
│           RUNTIME ENVIRONMENT                │
│           (Firecracker / gVisor)             │
│                                              │
│  ┌───────────────────────────────────────┐   │
│  │               AGENT                   │   │
│  │  Model · Tools · Memory               │   │
│  │  State · Harness                      │   │
│  └───────────────────────────────────────┘   │
└──────────────────────┬───────────────────────┘
                       │ runs on
┌──────────────────────▼───────────────────────┐
│           AGENT INFRASTRUCTURE               │
│                                              │
│  Kubernetes · VPC · NATS · Observability     │
│  Terraform · Cloud resources                 │
└──────────────────────┬───────────────────────┘
                       │ provisioned on
┌──────────────────────▼───────────────────────┐
│              CLOUD / COMPUTE                 │
│           GCP · AWS · Decentralized          │
└──────────────────────────────────────────────┘

Observability and Governance

Observability and governance span every layer. An agent that isn't observable isn't operable. An agent without governance isn't deployable in production.

Trace-level observability — every agent run is a trace. Every step — model call, tool invocation, memory read, memory write — is a span. LangSmith, Langfuse, AgentOps, and OpenTelemetry-based systems provide this.

Behavioral observability — understanding what agents decide and why. What reasoning patterns lead to successful outcomes? Where do agents consistently fail?

Cost observability — a single agent run might make 20 LLM calls across 3 models. Without per-run cost tracking, agent infrastructure costs become unpredictable.

Human-in-the-loop gates — for high-stakes decisions, the agent pauses and waits for human approval. These gates are designed into the orchestration layer as first-class components.

Guardrails — output validation preventing agents from taking actions outside defined bounds. NeMo Guardrails and Guardrails AI are production options.

Rate limits and circuit breakers — prevent runaway agent loops from consuming unbounded resources.

Audit trails — every agent action logged with identity, reasoning, inputs, outputs, and timestamp. Non-negotiable for production systems handling user data or financial transactions.

Where Agents Break in Production

Understanding the stack tells you exactly where failures happen:

Sandbox cold starts — the agent takes seconds to start because runtime initialization isn't optimized. Fix: memory snapshotting, warm pool of pre-initialized sandboxes.

Memory retrieval quality — the agent has relevant context but retrieves the wrong chunks. Fix: improve chunking strategy, add reranking, evaluate retrieval separately from generation.

Tool call governance gaps — the agent accesses data or systems it shouldn't. Fix: enforce RBAC at the tools layer, audit every tool call, use credential proxies.

Orchestration state loss — a long-running agent task fails and loses all progress. Fix: durable execution infrastructure (Inngest, Temporal), explicit checkpointing at each step.

Model cost explosion — the agent makes far more LLM calls than expected. Fix: cost observability, per-run budgets, circuit breakers on token consumption, model routing to cheaper models for simpler steps.

Behavioral blind spots — the agent is failing silently on a class of inputs and nobody knows. Fix: trace-level observability (LangSmith, Langfuse), systematic evaluation, behavioral monitoring.

Each failure mode maps to a specific layer. Engineers who understand the stack diagnose failures in minutes. Engineers who don't spend hours guessing.

The Outer Loop — What Engineers Actually Own

The stack tells you what the system is. The outer loop tells you what you're responsible for.

AI agents run the inner execution loop — investigate, implement, verify, repeat. The engineer owns the outer loop: the accountability boundary for what ships.

OUTER LOOP (engineer owns)
  ├── Constraints — inputs, architectures, instructions, invariants
  ├── Sampling — how much output to review
  ├── Audit — what evidence to keep
  └── Ownership — what part of the production boundary you sign your name on

    INNER LOOP (agent runs)
      ├── Investigate — gather context
      ├── Implement — produce the artifact
      ├── Verify — check against quality signals
      └── Repeat

At the production boundary: Quality → Evidence → Verdict — ship, block, or redirect. The human owns the verdict.

Creation is cheap; review is scarce. As of 2026, 42% of committed code is AI-generated. The bottleneck has shifted from "can we build this?" to "should this exist, and can we answer for it?"

Back pressure regulates autonomy. Not maximum autonomy — just enough to get the work done, with enough signal to stop, check, or correct.

Taste is an engineering asset. When anyone can generate anything, choosing what to make matters more. Skills get you leverage; accountability turns leverage into trust.

Summary

| Layer | What it does | Examples | | --- | --- | --- | | Model | Reasons and decides | GPT-4o, Claude, Llama | | Agent | Full unit of intelligence | Model + tools + memory + state + harness | | Orchestration | Coordinates multiple agents and workflows | LangGraph, CrewAI, custom loops | | Execution | Runs agents on real compute, manages lifecycle | Inngest, Temporal, custom platforms | | Runtime | Isolated sandbox per execution | Firecracker, gVisor, E2B, Modal | | Infrastructure | Platform everything runs on | K8s cluster, NATS, Prometheus | | Cloud / Compute | Physical or virtual machines | GCP, AWS, decentralized networks |

The agent isn't the stack. The agent is one layer in it.

Building this infrastructure — not just using it — is the frontier of AI engineering in 2026.

Next in the series: The Model Layer — training, inference, fine-tuning, and the shift from frontier APIs to open-source models.

Find me on Twitter or LinkedIn.