When to reach for it
- Large inputs can be split into independent chunks.
- Aggregation logic is clearly definable.
- Throughput scalability is critical.
Map-Reduce.
Map over chunks. Reduce to one answer.
A large task is mapped over independent chunks and subsequently aggregated or reduced into a single result.
When it backfires
- Global dependencies exist between chunks.
- The reduction would lose semantic value.
- A central context is required.
The tradeoff
Excellent scaling capabilities are achieved against the risk of generating inconsistent partial results.
What it actually does.
Input is split, mapped in parallel, then reduced.
Two ways this pattern will hurt you.
Reducer loses cross-chunk context
The reducer only sees per-chunk outputs and misses relationships that span chunk boundaries.
Fix · Include chunk metadata (position, overlap) in mapper output, or use a two-pass reducer that first reconciles boundaries.
Uneven chunk sizes overload one mapper
One chunk is 10x larger than the others. Its mapper dominates latency and the whole job waits.
Fix · Use dynamic chunk sizing or a size-aware splitter that balances load across mappers.
Where Map-Reduce is native.
Microsoft Agent Frameworkfan-out / fan-inNative
Anthropic Cookbookdocumented as a recipeAdaptable
Next pattern