Reproduce “Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wilson Y. Lee
This research identifies canonical path deviation as a causal mechanism for reliability failures in large language model agents performing long-horizon tasks. It demonstrates that stochastic drift from an empirically derived optimal solution path, rather than a lack of capability, causes agents to fail even when they are otherwise capable, with successful runs exhibiting a +0.060 Jaccard adherence gap.
Key insights
- Many agent failures are reliability failures, not capability failures, stemming from stochastic drift away from a task's inherent canonical solution path.
- Canonical path deviation is a gradual, self-reinforcing process rather than a single early mistake, with off-canonical calls increasing the probability of subsequent off-canonical calls by +22.7 percentage points.
- The variance in canonical path adherence provides an empirical taxonomy of failure: low variance for always-succeed, intermediate for always-fail (capability gap), and high for mixed-outcome units (reliability failure).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.