Reproduce “ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeremy Canale, Fabien Soulis
ReplayRCA evaluates whether a reviewer’s replacement action improves an agent’s chance of completing a task. It restores the environment before the proposed step, compares replacement and original actions across five continuations each, and scores the difference in success rates. Across 62 failures, Grok 4.7 achieved +51.3 percentage points; model rankings were unresolved, and results come from one simulated environment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.