#03
SQLite Under Concurrency
SymptomWrite locks serialize everything. Timeouts under load. Session bleeding.
FixAsyncPostgresSaver. Compose thread IDs as user×session.
L4 · Production
Memory, tools, runtime, governance, evals.
This is the floor every other rung was already standing on. We present L4 last because skipping it is the #1 reason agent prototypes die on contact with Tuesday — but conceptually, it is the foundation, not a final step.
The substrate
Every rung was leaning on this the whole time.
L1, L2, and L3 each pull from the operational layer beneath them. The spine is L4; each floor above annotates the L4 patterns it depends on.
Anatomy
What L4 is made of.
Five layers — Memory, Tools, Runtime, Governance, Observability — each a subdomain with its own primitives, its own failure modes, and its own answer to a different operational question.
Memory · What persists
Working state, long-term memory, vectors, checkpoints.
The pieces that survive past a single turn or a single run. Conversational, episodic, semantic, working — each with its own access pattern and bleed risk.
Tools · How the system acts
Registry, MCP, A2A, sandboxes.
The surface through which an agent reaches into the world. A registry catalogs capabilities; MCP and A2A standardize transport; sandboxes contain the blast radius.
Runtime · How it runs
DAGs, schedulers, durable execution, retries.
What makes a workflow resumable, restartable, and idempotent. The infrastructure layer that turns a graph into a service.
Governance · What it won't do
HITL gates, audit trails, schema validation.
The constraints that decide what gets through and what gets stopped. Where the system answers to people other than its operators.
Observability · How you see inside
Traces, evals, judges, cost tracking.
The telemetry that turns "it broke" into a diagnosis. Without this layer, every other layer is a black box.
Subdomains & patterns
Five layers. 34 patterns.
Each subdomain is a different operational question — memory, tools, runtime, governance, observability — and carries the production patterns that answer it. Together they are what L1, L2, and L3 were leaning on the whole time.
Memory architecture
What persists.
How conversation history, completed episodes, and structured knowledge are stored and retrieved across turns, runs, and sessions.
Where L3 systems leak across users and L2 workflows accumulate stale context. The choice between shared scratchpad and per-agent state is decided here.
Conversational Memory
The running message history of a session, kept in context so the agent can reference prior turns.
LangGraphOpenAI Agents SDKAutoGen / AG2Microsoft Agent Framework+1Episodic Memory
Completed task episodes — input, plan, outcome — persisted so the agent can recall similar cases.
LangGraphCrewAIAutoGen / AG2Semantic / Vector / Graph Memory
Long-term factual knowledge indexed in a vector or graph store for similarity retrieval.
LangGraphLlamaIndexMicrosoft Agent FrameworkWorking Memory / Scratchpad
A short-lived per-task buffer the agent uses while reasoning. Wiped between runs.
LangGraphOpenAI Agents SDKCrewAIVirtual Context Management
The agent pages facts between a resident context tier and an external store, like an OS swapping RAM and disk.
LettaMem0ZepLangChain / LlamaIndexRecorder
Captures every observation and tool result so a run can be replayed deterministically.
LangGraphLangSmithSkill Build
Extracts repeatedly-used procedures into reusable skills the agent can invoke by name.
Voyager-stylecustom
Tool integration
How it acts.
The surface through which the agent reaches into the world — function calling as the primitive, MCP and A2A as the transport, registries and adapters for scale.
Every L1 ReAct loop and every L3 specialist eventually sits behind a tool registry. Without one, the tool surface explodes and selection accuracy collapses.
Function Calling
The primitive — the model emits structured JSON and the harness invokes a typed function.
OpenAIAnthropicAll major SDKsTool Registry
A central catalog of tools alongside schemas, permissions, and versions. The antidote to tool explosion.
LangGraphOpenAI Agents SDKMicrosoft Agent FrameworkGoogle ADK+1MCP
Model Context Protocol — Anthropic's open standard for exposing tools and data sources as a uniform surface.
AnthropicAll major frameworksA2A Protocol
Agent-to-Agent — Google's open protocol for cross-framework agent discovery and messaging.
Google ADKMicrosoft Agent FrameworkAdapter Pattern
Wraps a legacy API or library in a thin tool-shaped interface the model can call.
AllCapability Routing
Selects which tool to invoke by capability tag rather than by name. Decouples caller and provider.
LangGraphMicrosoft Agent FrameworkPermission-scoped Tools
Tools that carry an explicit permission tag enforced at the harness, not the prompt.
LangGraphOpenAI Agents SDKAgentic RAG
The agent decides whether, when, and how to retrieve — reformulating queries and grading results — instead of retrieving once unconditionally.
LangGraphLlamaIndex
Runtime architecture
What makes it durable.
The infrastructure that turns a graph into a service: DAG execution, checkpointing, retries, actor mailboxes, event buses.
L2 workflows die without checkpointing. L3 multi-agent systems can't scale without an actor model or a pub/sub mesh. Production demands resumability.
Workflow DAG / Durable Execution
The graph runtime. Nodes, edges, retries, timeouts as first-class objects.
LangGraphTemporalAWS Step FunctionsCheckpointing / Resumability
Runtime persists state at every node boundary so a workflow can resume after crash or pause.
LangGraphActor Model
Each agent is an actor with its own mailbox and supervisor — concurrency without shared state.
RayAkka-stylecustomEvent-driven Choreography
Agents publish events to a bus; subscribers react. No central orchestrator decides.
KafkaNATSAWS StrandsSaga / Compensation
Long-running multi-step actions with explicit compensation handlers for partial failure.
LangGraphTemporalPub/Sub Agent Mesh
Agents discover each other through a broker; messages route by topic.
KafkaNATScustom
Governance & safety
What it won't do.
The constraints, gates, and validators that decide what gets through and what gets stopped. The layer that answers to people other than the operators.
HITL gates pause for approval. Sandboxes contain blast radius. Schema validation rejects malformed state. Audit trails make every decision non-repudiable.
Human-in-the-Loop Approval Gate
A workflow node that pauses for human review before continuing. Pause can last seconds or weeks.
LangGraphTemporalSandbox Execution
Untrusted code runs in a sealed environment so side-effects can't escape. Where CodeAct actually executes.
PyodidegVisorDockerE2BOutput Validation / Schema Enforcement
Pydantic-style validation on every node output. Reject and retry on schema failure; never inherit corrupted state.
LangGraphPydantic AIAllLeast Privilege Agent
Each agent gets the minimal tool surface its role requires. The antidote to permission sprawl.
AllAudit Trail
Append-only log of every decision, tool call, and message. Non-repudiable across time.
LangSmithcustomController
Polices the agent's actions against policy. Stops or modifies calls that violate rules.
LangGraphcustomIntegrator
Gathers and harmonizes outputs from multiple specialists before downstream propagation.
LangGraphcustomMultimodal Guardrails
Filters and classifiers on inputs and outputs across text, image, and audio modalities.
Llama GuardNeMo GuardrailscustomStatistical Guardrails
Numeric reject/allow gates: cosine-distance z-score for semantic drift, token entropy for low-confidence fabrication.
customNeMo GuardrailsGuardrails AI
Observability & evaluation
How you see inside.
The telemetry that turns 'it broke' into a diagnosis: traces, judges, evals, cost accounting. The difference between a debuggable system and one you can only rerun.
Without distributed tracing, every L3 failure is archaeology. Without evals, you can't tell if a prompt change made things better or worse.
Distributed Tracing
Every span — LLM call, tool call, handoff — recorded into a single trace tree across all agents.
LangSmithOpenTelemetryArizeToken / Cost Tracking
Per-call token and dollar accounting. Budgets enforced at the harness level, not after the fact.
LangSmithAllLLM-as-Judge
An LLM scores another LLM's output against a rubric. The score becomes a typed signal a workflow can route on.
LangSmithRagascustomIntegration Tests for Agents
Replayable test suites that exercise the agent end-to-end against fixtures and recorded traces.
LangSmithPromptfoo
Anti-patterns
Ways this rung goes wrong.
#07
Cascading Security Failures
from L3SymptomA poisoned doc in a shared index contaminates every downstream consumer. Prompt injection propagates via handoffs.
FixTreat every agent-to-agent message as a trust boundary. Partition knowledge bases by trust level.
Frameworks
What you build this on.
Deep dive
The last rung is also the spine.
You have just climbed the ladder, and the rung at the top doubles as the spine that runs down the side of every rung below it. Every L1 agent you've written has already pulled patterns from L4: it kept working memory in a context window, it called tools through some kind of registry, it ran on a runtime, and — if you were diligent — it emitted traces. The five subdomains named on this page are not new territory; they are what L1, L2, and L3 have been leaning on the whole time.
The ladder presents L4 last for a single pedagogical reason: skipping it is the most common cause of agent prototypes dying on contact with production. A reader told about L4 first but not about L1 has no concrete system in which production concerns matter. A reader who has built an L3 multi-agent system without memory architecture, sandboxes, HITL gates, or distributed tracing will learn the hard way that the spine was always load-bearing.
There is no “L5.” The work of L4 is to push the operational discipline of this layer sideways into L1, L2, and L3 — to treat memory architecture, tool surfaces, runtime guarantees, governance, and observability as concerns of the system from the first prompt onward, not concerns to bolt on once it works in a demo. The discipline of treating production as a spine that touches every rung is what survives Tuesday.
You've reached the floor
The ladder isn't a hierarchy — it's a loop.
L4 isn't the summit; it's the ground every other rung stands on. A production system threads all four: an L1 loop, wired into L2 flow, coordinated as L3 specialists, running on the L4 substrate. Climb back down and read each rung knowing what holds it up.
References
Where this rung was first written down.
- 1973Hewitt, Bishop & Steiger — A Universal Modular Actor Formalism for Artificial Intelligence
- 1987Garcia-Molina & Salem — Sagassource
- 1978Lamport — Time, Clocks, and the Ordering of Events in a Distributed System
- 2004Dean & Ghemawat — MapReduce: Simplified Data Processing on Large Clusterssource
- 2010Sigelman et al. — Dapper, a Large-Scale Distributed Systems Tracing Infrastructuresource
- 2024Anthropic — Model Context Protocol (MCP) — Specificationsource
- 2024Google — Agent-to-Agent (A2A) Protocol
- 2024LangChain — LangGraph: Building Stateful, Multi-Actor Applications with LLMssource
- 2024Anthropic — Building Effective Agentssource