Multi‑Agent Systems

Steps you wire in code. The LLM fills the slots.

Structure belongs in code. Workflow patterns are the deterministic skeleton: sequential, routing, parallel, loops. The model produces content; you control flow.

Open Sequential Pipeline, the canonical primitive See the four primitives composed

Anatomy

What an L2 workflow is made of.

Five parts compose every workflow on this rung: Step, Edge, Router, Joiner, and a Loop control gate. The diagram shows one canonical graph; the cards below decompose each part.

INPUTTypedSTEPLLM · typed I/OROUTERBranch by classABBRANCH ASpecialistBRANCH BSpecialistJOINERFan-inOUTPUTExitloop · until budget / signal

  • Step · The unit of work

    One node, typed in and out.

    A node that calls an LLM (or a tool) with typed inputs and outputs. Same contract every time it runs.

  • Edge · The deterministic line

    Reachable, ordered, no surprises.

    A directed connection from one node to another. The set of edges defines what's possible; the runtime never invents new ones.

  • Router · The branch

    Switch on intent or schema.

    A decision node that dispatches by classification, schema, or rule. The branch logic is in code; the model only emits the signal.

  • Joiner · The fan-in

    Parallel branches collapse here.

    Where two or more concurrent branches meet and their outputs merge back into a single state shape.

  • Loop control · The exit clause

    Stop the repetition.

    Budget, threshold, or signal that ends a loop. Without one, an L2 system has an unbounded-loop anti-pattern waiting to happen.

When this rung fits

Workflow is enough when…

A workflow earns its name when the control flow is in code and the LLM only fills the slots. Below: the signals to commit, and the signals to climb.

A single agent is enough when…
  • You can sketch the control flow on a whiteboard before the LLM gets involved.
  • The branches are knowable: a router has a fixed list of options.
  • Failures need to be localisable — "step 3 broke" is more useful than "the agent broke".
  • Latency and token budget matter: predictable shape lets you cache, parallelize, and cap.
  • You need typed inputs/outputs at every node boundary for downstream systems.
…and no longer enough when
  • Which agent should speak next is itself the decision the system has to make — go to L3.
  • The set of possible next steps cannot be enumerated at authoring time.
  • A single adaptive loop is enough and the orchestration adds no value — drop to L1.
  • Steps are highly conditional on previous outputs in ways you can't pre-specify — drop to L1's ReAct.

Primitives

Four shapes. Compose anything.

Anthropic's Building Effective Agents names these as the workflow alphabet. Most L2 systems compose two or three primitives together; the variants below the cards (orchestrator-workers, evaluator-optimizer, etc.) are compositions you'll recognize at scale.

  • Sequential

    One step's output is the next step's input.

    The simplest legible shape. Predictable, debuggable, ordered.

  • Routing

    A decision node dispatches by class.

    Where different requests need different handling and the classes are knowable.

  • Parallel

    Independent branches fan out, joiner fans them back in.

    Cut latency, or vote across N runs for robustness. Workers never coordinate.

  • Loop

    Repeat until a budget, threshold, or signal — never until the model decides.

    The foundation every quality loop (evaluator-optimizer, iterative-refinement) is built on.

What it runs on

How an L2 workflow persists.

The four primitives are graphs. Graphs need a runtime that can checkpoint state, resume after failure, pause for humans, and judge intermediate quality. These four L4 primitives are the foundation every long-running L2 system runs on.

  • Checkpointing

    Resumable Workflow · State Snapshot

    Runtime Architecture

    The runtime persists state at every node boundary so a workflow can be resumed, replayed, or branched without re-running prior steps.

    Why for L2Long L2 workflows die without it. A 12-step pipeline that crashes on step 11 should not restart from step 1.

    See in L4

  • Workflow DAG

    Durable Execution · DAG Orchestration

    Runtime Architecture

    The graph runtime L2 sits on. Nodes and edges as first-class runtime objects with retries, timeouts, and state history.

    Why for L2Sequential, Routing, Parallel, Loop — every L2 primitive is a shape the DAG runtime materializes for you.

    See in L4

  • HITL Gate

    Human-in-the-Loop · Approval Step

    Governance & Safety

    A workflow node that pauses for human review before continuing. Pause can last seconds or weeks; checkpointing keeps it cheap.

    Why for L2HITL is a control overlay that composes with every rung. L2 is where it lands most legibly — a deterministic pause point in a deterministic graph is the easiest seam between automatic and human steps to audit.

    See in L4

  • LLM-as-Judge

    AI Critic · Automated Evaluator

    Observability & Evaluation

    An LLM scores another step's output against a rubric. The score becomes a typed signal the workflow can route on.

    Why for L2Evaluator-Optimizer and Iterative Refinement are LLM-as-Judge composed with a Loop. Without it, neither pattern works.

    See in L4

Patterns · 9 on this rung

9 workflow patterns on this rung.

Each card is a recurring way to wire deterministic flow around non-deterministic model calls. Scan use-when / don't / trade-off; open one to deep-dive.

Anthropic primitives

The four shapes Anthropic's "Building Effective Agents" names as the workflow alphabet.

  • Sequential Pipeline

    aka Prompt Chaining · Linear Workflow · Sequential Process

    Open

    Multiple steps are executed in a fixed order, where each step utilizes the output of the previous one.

    Use when
    • The task naturally breaks down into phases, each delivering a verifiable intermediate product.
    • Control is more important than autonomy.
    Don't
    • The workflow branches heavily or results dynamically generate new goals.
    • Steps can be parallelized without dependencies.
    Trade-off
    High control-flow predictability is gained at the cost of low flexibility; output content is still non-deterministic at every LLM node.
    CrewAIGoogle ADKMicrosoft Agent FrameworkAWS Strands+1

    Wu et al. (2022) — AI Chains: Transparent Human-AI Interaction by Chaining LLM Prompts

  • Routing

    aka Classifier Router · Intent Routing · Conditional Branching

    Open

    A classification or decision module classifies a request and dispatches it to specific targets, such as a specialized prompt, agent, or tool.

    Use when
    • Different request classes require distinct handling.
    • Specialized tools exist for specific intents.
    • Misrouting carries a high cost.
    Don't
    • All tasks use the same flow.
    • The classification is unstable.
    • The routing logic becomes more complex than the task itself.
    Trade-off
    More precise handling is achieved at the expense of additional decision logic and the risk of misclassification.
    LangGraphGoogle ADKMicrosoft Agent FrameworkAWS Strands+1

    Anthropic (2024) — Building Effective Agents § Routing

  • Parallelization

    aka Sectioned Parallelism · Ensemble Voting · Fan-out · Divide and Process

    Open

    Independent subtasks are processed in parallel and either merged (sectioning) or the best result is selected via an aggregator (voting).

    Use when
    • The input is naturally segmentable and tasks are independent to reduce latency (sectioning).
    • Robustness is more important than single execution costs (voting).
    Don't
    • Strong dependencies exist between segments.
    • Semantic merging is difficult.
    • Latency and token budgets are strictly limited.
    Trade-off
    Lower latency and higher robustness are gained against increased integration complexity and multiple execution costs.
    Google ADKMicrosoft Agent FrameworkAWS StrandsLangGraph+1

    Anthropic (2024) — Building Effective Agents § Parallelization

  • Loop

    aka Control Loop · Retry Loop · Agent Loop

    Open

    One or more steps are repeated until a specific budget, quality bound, or exit condition is reached.

    Use when
    • The result can be iteratively improved.
    • External validation triggers a retry.
    • Tool results necessitate new iterations.
    Don't
    • There is no stable exit condition.
    • Costs can spiral out of control.
    • Errors amplify through repetition.
    Trade-off
    Adaptive improvement is achieved at the risk of endless or highly expensive execution loops.
    LangGraphCrewAIMicrosoft Agent FrameworkGoogle ADK+1

    Anthropic (2024) — Building Effective Agents § Iterative Refinement

Orchestration variants

Compositions of the primitives that recur often enough to earn their own names.

  • Orchestrator-Workers

    aka Coordinator-Workers · Manager-Worker · Dynamic Task Decomposition · Multi-Agent Collaboration · Coordinating Agent

    Open

    An orchestrator dynamically decomposes a task and assigns subtasks to specialized workers, managing the aggregation centrally.

    Use when
    • Subtasks only become apparent at runtime.
    • Workers handle highly specialized functions.
    • Aggregation must remain centrally controlled.
    Don't
    • A static workflow suffices.
    • Workers lack clear responsibilities.
    • The orchestrator becomes a severe bottleneck.
    Trade-off
    Highly flexible delegation is gained against significant coordination and integration overhead.
    LangGraphCrewAIGoogle ADKMicrosoft Agent Framework+2

    Anthropic (2024) — Building Effective Agents § Orchestrator-Workers

  • Map-Reduce

    aka Fan-out/Fan-in · Map Aggregate · Batch Decomposition

    Open

    A large task is mapped over independent chunks and subsequently aggregated or reduced into a single result.

    Use when
    • Large inputs can be split into independent chunks.
    • Aggregation logic is clearly definable.
    • Throughput scalability is critical.
    Don't
    • Global dependencies exist between chunks.
    • The reduction would lose semantic value.
    • A central context is required.
    Trade-off
    Excellent scaling capabilities are achieved against the risk of generating inconsistent partial results.
    LangGraphAWS StrandsGoogle ADKMicrosoft Agent Framework+1

    Dean & Ghemawat (2004) — MapReduce: Simplified Data Processing on Large Clusters

  • Resource-Aware Optimization

    aka Model Routing · Cost-Aware Routing · LLM Router · Model Cascade

    Open

    A router scores each request's complexity and dispatches it to the cheapest model that still meets the quality bar, under explicit token / time / cost budgets, using prompt caching and model cascades.

    Use when
    • Request volume is high with mixed difficulty.
    • Cost is a first-order constraint.
    • A cheaper model handles a meaningful share of traffic without quality loss.
    Don't
    • Every task needs the top model.
    • The complexity rubric is unstable — mis-route cost exceeds the savings.
    • The latency of an extra routing hop is unacceptable.
    Trade-off
    Large cost savings are gained against the effort of tuning and continuously validating the complexity rubric.
    RouteLLMOpenRouterLangGraph

    Ong et al. (2024) — RouteLLM: Learning to Route LLMs with Preference Data

Quality loops

Iteration shapes for improving a single artifact.

  • Evaluator-Optimizer

    aka Generator-Critic · Critique and Revise · Evaluate-Improve

    Open

    A generator produces a result, an evaluator scores it against specific criteria, and the generator optimizes it based on the feedback.

    Use when
    • Quality criteria can be explicitly formulated.
    • Iterative improvement is measurable.
    • Creative outputs need rigorous checking.
    Don't
    • The evaluator cannot provide reliable signals.
    • Schema validation suffices.
    • The budget does not allow for multiple runs.
    Trade-off
    Better output quality is gained against additional evaluation complexity and latency.
    Google ADKLangGraphAutoGen / AG2Microsoft Agent Framework+1

    Madaan et al. (2023) — Self-Refine: Iterative Refinement with Self-Feedback

  • Iterative Refinement

    aka Revise Loop · Draft-Improve · Progressive Refinement

    Open

    Controlled passes improve a single artifact across revisions, often using explicit feedback from rules, tests, or users.

    Use when
    • Result quality increases gradually and intermediate states must be preserved.
    • External feedback is readily available.
    Don't
    • A valid result is typically generated in one step.
    • Revisions lack clear signals.
    • Consistency drops through repeated rewriting.
    Trade-off
    Improved artifact quality is achieved at the expense of longer runtimes and potential thematic drift.
    Google ADKLangGraphCrewAIMicrosoft Agent Framework

    Madaan et al. (2023) — Self-Refine: Iterative Refinement with Self-Feedback

Anti-patterns

Ways this rung goes wrong.

  • #05

    Hallucinated Routing

    SymptomReflection loops never converge. Recursion limits trip at runtime instead of design time.

    FixSchema-validate router decisions. Hard recursion_limit, hard max_handoffs.

  • #08

    Unbounded Loop

    from L1

    SymptomAn L1 agent stuck in unproductive infinite cycles due to flawed reasoning, with no programmed recursion limit or step budget.

    FixSet a hard recursion_limit and a step budget at the harness level. Add a Reflexion gate or a tool-call counter to break runaway loops.

    Originated on L1 with ReAct; recurs here whenever a primitive Loop has no termination.

  • #01

    Over-Agentification

    from L3

    SymptomHigh latency, expensive token bills, debugging that requires reconstructing emergent behavior from logs.

    FixWalk the decision heuristic. Sketch the simpler alternative side-by-side before you commit.

    Mirrors the L3 anti-pattern: reaching for multi-agent when an L2 workflow would have sufficed.

  • #03

    SQLite Under Concurrency

    from L4

    SymptomWrite locks serialize everything. Timeouts under load. Session bleeding.

    FixAsyncPostgresSaver. Compose thread IDs as user×session.

    An L4 anti-pattern that bites L2 hardest — every checkpointed workflow eventually outgrows SQLite.

Frameworks

What you build this on.

LangGraphpython
from typing import TypedDict
from langgraph.graph import StateGraph, END

class State(TypedDict):
    raw: str
    parsed: dict | None
    analysis: dict | None
    payload: dict | None
    answer: str | None

def ingest(state):    return {"parsed": parse(state["raw"])}
def analyze(state):   return {"analysis": analyze_llm(state["parsed"])}
def synthesize(state): return {"payload": synth_llm(state["analysis"])}
def respond(state):   return {"answer": render_md(state["payload"])}

g = StateGraph(State)
g.add_node("ingest", ingest)
g.add_node("analyze", analyze)
g.add_node("synthesize", synthesize)
g.add_node("respond", respond)
g.add_edge("ingest", "analyze")
g.add_edge("analyze", "synthesize")
g.add_edge("synthesize", "respond")
g.add_edge("respond", END)
g.set_entry_point("ingest")
pipeline = g.compile()

Deep dive

Structure belongs in code.

An L2 system is a workflow whose structure is fixed in code and whose slots are filled by an LLM at each step. The control flow is deterministic — you can read it, draw it, reason about it without running it. The non-determinism lives inside individual node calls, not in the path between them. Anthropic's "Building Effective Agents" frames this as the trade between agency and predictability: the fewer choices the LLM gets to make about which node runs next, the more reliable the system as a whole.

The four primitives — sequential, routing, parallel, loop — compose into the variants you'll recognize at scale. Orchestrator-workers is a routing pattern with a join. Map-reduce is parallel-with-aggregator. Evaluator-optimizer is a loop with a generator and a critic. The vocabulary differs across frameworks; the topology does not.

You outgrow L2 when the next branch can't be expressed as a switch — when which agent should speak is itself the decision the system has to make. At that point the workflow's edges become too numerous or too context-dependent to enumerate, and the natural move is to L3, where a coordinator decides routing at runtime instead of you deciding it at authoring time.

The pattern here most often mistaken for L3 is Orchestrator-Workers: it has a central node that delegates, so it reads as multi-agent. It stays L2 because the orchestrator is a control structure you authored — the decompose → fan-out → aggregate shape is fixed in code, and the workers are ephemeral, task-scoped calls. Its L3 twin, Supervisor, keeps persistent specialist agents and lets a coordinator agent decide routing at runtime: same silhouette, different rung.

When to climb

When the next step is itself a decision, add specialists.

A workflow's edges are fixed at authoring time. The moment "which agent should act next" becomes the actual problem — when specialists genuinely differ in prompt, tools, and trust scope — you've reached L3: coordination between separate minds.

Climb to L3 — Multi-agent

References

Where this rung was first written down.

The L1–L4 ladder is a didactic construct of this project — a way to sequence the catalog, not an external model. Its workflow-vs-agent seam (L2↔L3) follows Anthropic's Building Effective Agents (2024); each rung's patterns carry their own primary references.

Search

Search patterns, frameworks, and pages.