Reproduce “Knowing When to Intervene: Causal Evaluation of AI Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ayush Sawarni, Wenqian Xing, Tijana Zrnic, Ramesh Johari, Vasilis Syrgkanis
The authors evaluate intervention effects during agent runs by randomizing review or control at eligible decisions. For Qwen3.5 Flash on ImpossibleBench, adversarial feedback after the fifth failed attempt, with one attempt left, increased estimated terminal detected rule violations by 9.4 percentage points, versus 3.3 after the first failure. These are assignment effects in this experiment, not universal predictions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.