When to reach for it
- The problem has multiple plausible solution paths.
- Early decisions have high cascading effects.
- The search space can be meaningfully evaluated by the LLM.
Tree of Thoughts.
Branch the reasoning. Keep the best leaf.
The agent explores multiple reasoning paths simultaneously like a tree diagram and selects promising paths to pursue while discarding dead ends.
When it backfires
- The task is linear or directly solvable.
- Costs and latency must remain low.
The tradeoff
Enables a much broader and deeper exploration of the solution space, but leads to exponentially increasing computational and token overhead.
A shape you can draw on a napkin.
Each thought spawns a few children; promising branches expand, dead ends stop.
Two ways this pattern will hurt you.
Branching factor blows up the budget
Naively expanding every node multiplies LLM calls exponentially without obvious payoff.
Fix · Cap branching factor and depth; prune by score after each layer.
Pruning heuristic is also an LLM
The pruner is itself non-deterministic; on tight budgets it can discard the best branch.
Fix · Anchor pruning to a verifiable signal (executable check, oracle, regex) when possible.