Reproduce “A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wendy Zheng, Liang Wu, Yinhan He, Mayank Darbari, Jundong Li, Liangjie Hong

The authors study automated failure attribution, which identifies the earliest erroneous step in a multi-agent transcript. They compare prompting, linear probes on model activations, and grafting, an intervention that routes an internal signal toward the output. On Who&When, grafting improves step accuracy over prompting, including 38.10% versus 23.81% for Qwen 2.5 0.5B on Algo.; probes remain stronger in many settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.