Multi‑Agent Systems

/ladder/l2/

L2 · Workflow

Steps you wire in code. The LLM fills the slots.

Structure belongs in code. Workflow patterns are the deterministic skeleton: sequential, routing, parallel, loops. The model produces content; you control flow.

Open Sequential Pipeline, the canonical primitive See the four primitives composed

Anatomy

What an L2 workflow is made of.

Five parts compose every workflow on this rung: Step, Edge, Router, Joiner, and a Loop control gate. The diagram shows one canonical graph; the cards below decompose each part.

INPUTTypedSTEPLLM · typed I/OROUTERBranch by classABBRANCH ASpecialistBRANCH BSpecialistJOINERFan-inOUTPUTExitloop · until budget / signal

  • Step · The unit of work

    One node, typed in and out.

    A node that calls an LLM (or a tool) with typed inputs and outputs. Same contract every time it runs.

  • Edge · The deterministic line

    Reachable, ordered, no surprises.

    A directed connection from one node to another. The set of edges defines what's possible; the runtime never invents new ones.

  • Router · The branch

    Switch on intent or schema.

    A decision node that dispatches by classification, schema, or rule. The branch logic is in code; the model only emits the signal.

  • Joiner · The fan-in

    Parallel branches collapse here.

    Where two or more concurrent branches meet and their outputs merge back into a single state shape.

  • Loop control · The exit clause

    Stop the repetition.

    Budget, threshold, or signal that ends a loop. Without one, an L2 system has an unbounded-loop anti-pattern waiting to happen.

When this rung fits

Workflow is enough when…

A workflow earns its name when the control flow is in code and the LLM only fills the slots. Below: the signals to commit, and the signals to climb.

A single agent is enough when…
  • You can sketch the control flow on a whiteboard before the LLM gets involved.
  • The branches are knowable: a router has a fixed list of options.
  • Failures need to be localisable — "step 3 broke" is more useful than "the agent broke".
  • Latency and token budget matter: predictable shape lets you cache, parallelise, and cap.
  • You need typed inputs/outputs at every node boundary for downstream systems.
…and no longer enough when
  • Which agent should speak next is itself the decision the system has to make — go to L3.
  • The set of possible next steps cannot be enumerated at authoring time.
  • A single adaptive loop is enough and the orchestration adds no value — drop to L1.
  • Steps are highly conditional on previous outputs in ways you can't pre-specify — drop to L1's ReAct.

Primitives

Four shapes. Compose anything.

Anthropic's Building Effective Agents names these as the workflow alphabet. Most L2 systems compose two or three primitives together; the variants below the cards (orchestrator-workers, evaluator-optimizer, etc.) are compositions you'll recognise at scale.

  • Sequential

    One step's output is the next step's input.

    The simplest legible shape. Predictable, debuggable, ordered.

  • Routing

    A decision node dispatches by class.

    Where different requests need different handling and the classes are knowable.

  • Parallel

    Independent branches fan out, joiner fans them back in.

    Cut latency, or vote across N runs for robustness. Workers never coordinate.

  • Loop

    Repeat until a budget, threshold, or signal — never until the model decides.

    The foundation every quality loop (evaluator-optimizer, iterative-refinement) is built on.

What it runs on

How an L2 workflow persists.

The four primitives are graphs. Graphs need a runtime that can checkpoint state, resume after failure, pause for humans, and judge intermediate quality. These four L4 primitives are the foundation every long-running L2 system runs on.

  • Checkpointing

    Resumable Workflow · State Snapshot

    Runtime Architecture

    The runtime persists state at every node boundary so a workflow can be resumed, replayed, or branched without re-running prior steps.

    Why for L2Long L2 workflows die without it. A 12-step pipeline that crashes on step 11 should not restart from step 1.

    See in L4

  • Workflow DAG

    Durable Execution · DAG Orchestration

    Runtime Architecture

    The graph runtime L2 sits on. Nodes and edges as first-class runtime objects with retries, timeouts, and state history.

    Why for L2Sequential, Routing, Parallel, Loop — every L2 primitive is a shape the DAG runtime materialises for you.

    See in L4

  • HITL Gate

    Human-in-the-Loop · Approval Step

    Governance & Safety

    A workflow node that pauses for human review before continuing. Pause can last seconds or weeks; checkpointing keeps it cheap.

    Why for L2HITL is a control overlay that composes with every rung. L2 is where it lands most legibly — a deterministic pause point in a deterministic graph is the easiest seam between automatic and human steps to audit.

    See in L4

  • LLM-as-Judge

    AI Critic · Automated Evaluator

    Observability & Evaluation

    An LLM scores another step's output against a rubric. The score becomes a typed signal the workflow can route on.

    Why for L2Evaluator-Optimizer and Iterative Refinement are LLM-as-Judge composed with a Loop. Without it, neither pattern works.

    See in L4

Patterns · 9 on this rung

9 workflow patterns on this rung.

Each card is a recurring way to wire deterministic flow around non-deterministic model calls. Scan use-when / don't / trade-off; open one to deep-dive.

Anthropic primitives

The four shapes Anthropic's "Building Effective Agents" names as the workflow alphabet.

  • Sequential Pipeline

    aka Prompt Chaining · Linear Workflow · Sequential Process

    Open

    Multiple steps are executed in a fixed order, where each step utilizes the output of the previous one.

    Use when
    • The task naturally breaks down into phases, each delivering a verifiable intermediate product.
    • Control is more important than autonomy.
    Don't
    • The workflow branches heavily or results dynamically generate new goals.
    • Steps can be parallelized without dependencies.
    Trade-off
    High control-flow predictability is gained at the cost of low flexibility; output content is still non-deterministic at every LLM node.
    CrewAIGoogle ADKMicrosoft Agent FrameworkAWS Strands+1

    Wu et al. (2022) — AI Chains: Transparent Human-AI Interaction by Chaining LLM Prompts

  • Routing

    aka Classifier Router · Intent Routing · Conditional Branching

    Open

    A classification or decision module classifies a request and dispatches it to specific targets, such as a specialized prompt, agent, or tool.

    Use when
    • Different request classes require distinct handling.
    • Specialized tools exist for specific intents.
    • Misrouting carries a high cost.
    Don't
    • All tasks use the same flow.
    • The classification is unstable.
    • The routing logic becomes more complex than the task itself.
    Trade-off
    More precise handling is achieved at the expense of additional decision logic and the risk of misclassification.
    LangGraphGoogle ADKMicrosoft Agent FrameworkAWS Strands+1

    Anthropic (2024) — Building Effective Agents § Routing

  • Parallelization

    aka Sectioned Parallelism · Ensemble Voting · Fan-out · Divide and Process

    Open

    Independent subtasks are processed in parallel and either merged (sectioning) or the best result is selected via an aggregator (voting).

    Use when
    • The input is naturally segmentable and tasks are independent to reduce latency (sectioning).
    • Robustness is more important than single execution costs (voting).
    Don't
    • Strong dependencies exist between segments.
    • Semantic merging is difficult.
    • Latency and token budgets are strictly limited.
    Trade-off
    Lower latency and higher robustness are gained against increased integration complexity and multiple execution costs.
    Google ADKMicrosoft Agent FrameworkAWS StrandsLangGraph+1

    Anthropic (2024) — Building Effective Agents § Parallelization

  • Loop

    aka Control Loop · Retry Loop · Agent Loop

    Open

    One or more steps are repeated until a specific budget, quality bound, or exit condition is reached.

    Use when
    • The result can be iteratively improved.
    • External validation triggers a retry.
    • Tool results necessitate new iterations.
    Don't
    • There is no stable exit condition.
    • Costs can spiral out of control.
    • Errors amplify through repetition.
    Trade-off
    Adaptive improvement is achieved at the risk of endless or highly expensive execution loops.
    LangGraphCrewAIMicrosoft Agent FrameworkGoogle ADK+1

    Anthropic (2024) — Building Effective Agents § Iterative Refinement

Orchestration variants

Compositions of the primitives that recur often enough to earn their own names.

  • Orchestrator-Workers

    aka Coordinator-Workers · Manager-Worker · Dynamic Task Decomposition · Multi-Agent Collaboration

    Open

    An orchestrator dynamically decomposes a task and assigns subtasks to specialized workers, managing the aggregation centrally.

    Use when
    • Subtasks only become apparent at runtime.
    • Workers handle highly specialized functions.
    • Aggregation must remain centrally controlled.
    Don't
    • A static workflow suffices.
    • Workers lack clear responsibilities.
    • The orchestrator becomes a severe bottleneck.
    Trade-off
    Highly flexible delegation is gained against significant coordination and integration overhead.
    LangGraphCrewAIGoogle ADKMicrosoft Agent Framework+2

    Anthropic (2024) — Building Effective Agents § Orchestrator-Workers

  • Map-Reduce

    aka Fan-out/Fan-in · Map Aggregate · Batch Decomposition

    Open

    A large task is mapped over independent chunks and subsequently aggregated or reduced into a single result.

    Use when
    • Large inputs can be split into independent chunks.
    • Aggregation logic is clearly definable.
    • Throughput scalability is critical.
    Don't
    • Global dependencies exist between chunks.
    • The reduction would lose semantic value.
    • A central context is required.
    Trade-off
    Excellent scaling capabilities are achieved against the risk of generating inconsistent partial results.
    LangGraphAWS StrandsGoogle ADKMicrosoft Agent Framework+1

    Dean & Ghemawat (2004) — MapReduce: Simplified Data Processing on Large Clusters

  • Resource-Aware Optimization

    aka Model Routing · Cost-Aware Routing · LLM Router · Model Cascade

    Open

    A router scores each request's complexity and dispatches it to the cheapest model that still meets the quality bar, under explicit token, time, and cost budgets, using prompt caching and model cascades.

    Use when
    • Request volume is high with mixed difficulty.
    • Cost is a first-order constraint.
    • A cheaper model handles a meaningful share of traffic without quality loss.
    Don't
    • Every task needs the top model.
    • The complexity rubric is unstable — mis-route cost exceeds the savings.
    • The latency of an extra routing hop is unacceptable.
    Trade-off
    Large cost savings are gained against the effort of tuning and continuously validating the complexity rubric.
    RouteLLMOpenRouterLangGraph

    Ong et al. (2024) — RouteLLM: Learning to Route LLMs with Preference Data

Quality loops

Iteration shapes for improving a single artifact.

  • Evaluator-Optimizer

    aka Generator-Critic · Critique and Revise · Evaluate-Improve

    Open

    A generator produces a result, an evaluator scores it against specific criteria, and the generator optimizes it based on the feedback.

    Use when
    • Quality criteria can be explicitly formulated.
    • Iterative improvement is measurable.
    • Creative outputs need rigorous checking.
    Don't
    • The evaluator cannot provide reliable signals.
    • Schema validation suffices.
    • The budget does not allow for multiple runs.
    Trade-off
    Better output quality is gained against additional evaluation complexity and latency.
    Google ADKLangGraphAutoGen / AG2Microsoft Agent Framework+1

    Madaan et al. (2023) — Self-Refine: Iterative Refinement with Self-Feedback

  • Iterative Refinement

    aka Revise Loop · Draft-Improve · Progressive Refinement

    Open

    Controlled passes improve a single artifact across revisions, often using explicit feedback from rules, tests, or users.

    Use when
    • Result quality increases gradually and intermediate states must be preserved.
    • External feedback is readily available.
    Don't
    • A valid result is typically generated in one step.
    • Revisions lack clear signals.
    • Consistency drops through repeated rewriting.
    Trade-off
    Improved artifact quality is achieved at the expense of longer runtimes and potential thematic drift.
    Google ADKLangGraphCrewAIMicrosoft Agent Framework

    Madaan et al. (2023) — Self-Refine: Iterative Refinement with Self-Feedback

Anti-patterns

Ways this rung goes wrong.

  • #05

    Hallucinated Routing

    SymptomReflection loops never converge. Recursion limits trip at runtime instead of design time.

    FixSchema-validate router decisions. Hard recursion_limit, hard max_handoffs.

  • #08

    Unbounded Loop

    from L1

    SymptomAn L1 agent stuck in unproductive infinite cycles due to flawed reasoning, with no programmed recursion limit or step budget.

    FixSet a hard recursion_limit and a step budget at the harness level. Add a Reflexion gate or a tool-call counter to break runaway loops.

    Originated on L1 with ReAct; recurs here whenever a primitive Loop has no termination.

  • #01

    Over-Agentification

    from L3

    SymptomHigh latency, expensive token bills, debugging that requires reconstructing emergent behaviour from logs.

    FixWalk the decision heuristic. Sketch the simpler alternative side-by-side before you commit.

    Mirrors the L3 anti-pattern: reaching for multi-agent when an L2 workflow would have sufficed.

  • #03

    SQLite Under Concurrency

    from L4

    SymptomWrite locks serialize everything. Timeouts under load. Session bleeding.

    FixAsyncPostgresSaver. Compose thread IDs as user×session.

    An L4 anti-pattern that bites L2 hardest — every checkpointed workflow eventually outgrows SQLite.

Frameworks

What you build this on.

LangGraphpython
from typing import TypedDict
from langgraph.graph import StateGraph, END

class State(TypedDict):
    raw: str
    parsed: dict | None
    analysis: dict | None
    payload: dict | None
    answer: str | None

def ingest(state):    return {"parsed": parse(state["raw"])}
def analyze(state):   return {"analysis": analyze_llm(state["parsed"])}
def synthesize(state): return {"payload": synth_llm(state["analysis"])}
def respond(state):   return {"answer": render_md(state["payload"])}

g = StateGraph(State)
g.add_node("ingest", ingest)
g.add_node("analyze", analyze)
g.add_node("synthesize", synthesize)
g.add_node("respond", respond)
g.add_edge("ingest", "analyze")
g.add_edge("analyze", "synthesize")
g.add_edge("synthesize", "respond")
g.add_edge("respond", END)
g.set_entry_point("ingest")
pipeline = g.compile()

Deep dive

Structure belongs in code.

An L2 system is a workflow whose structure is fixed in code and whose slots are filled by an LLM at each step. The control flow is deterministic — you can read it, draw it, reason about it without running it. The non-determinism lives inside individual node calls, not in the path between them. Anthropic's "Building Effective Agents" frames this as the trade between agency and predictability: the fewer choices the LLM gets to make about which node runs next, the more reliable the system as a whole.

The four primitives — sequential, routing, parallel, loop — compose into the variants you'll recognize at scale. Orchestrator-workers is a routing pattern with a join. Map-reduce is parallel-with-aggregator. Evaluator-optimizer is a loop with a generator and a critic. The vocabulary differs across frameworks; the topology does not.

You outgrow L2 when the next branch can't be expressed as a switch — when which agent should speak is itself the decision the system has to make. At that point the workflow's edges become too numerous or too context-dependent to enumerate, and the natural move is to L3, where a coordinator decides routing at runtime instead of you deciding it at authoring time.

The pattern here most often mistaken for L3 is Orchestrator-Workers: it has a central node that delegates, so it reads as multi-agent. It stays L2 because the orchestrator is a control structure you authored — the decompose → fan-out → aggregate shape is fixed in code, and the workers are ephemeral, task-scoped calls. Its L3 twin, Supervisor, keeps persistent specialist agents and lets a coordinator agent decide routing at runtime: same silhouette, different rung.

When to climb

When the next step is itself a decision, add specialists.

A workflow's edges are fixed at authoring time. The moment "which agent should act next" becomes the actual problem — when specialists genuinely differ in prompt, tools, and trust scope — you've reached L3: coordination between separate minds.

Climb to L3 — Multi-agent

References

Where this rung was first written down.

The L1–L4 ladder is a didactic construct of this project — a way to sequence the catalog, not an external model. Its workflow-vs-agent seam (L2↔L3) follows Anthropic's Building Effective Agents (2024); each rung's patterns carry their own primary references.

Search

Search patterns, frameworks, and pages.