Reproduce “Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaineet Shah
Causal Agent Replay debugs harmful agent outcomes by resampling a step from the same policy, rerunning the continuation, and measuring outcome changes. It selects the latest step with a nonzero effect and adds Shapley attribution for interactions. In synthetic SCM tests it finds planted causes and assigns 0.44 and 0.45 to interacting causes, but real-agent accuracy remains untested.
Key insights
- Accurate failure attribution for LLM agents requires an intervention-based, causal framework rather than mere observation, to precisely isolate the true originating decision.
- The inherent stochasticity of LLM policies necessitates generating distributions over outcomes with confidence intervals following interventions to provide robust and reliable causal insights.
- Resolving confounds in stochastic replay, particularly the 'point-of-commitment' rule, is essential for localizing causal steps correctly when re-rolling downstream decisions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.