Reproduce “Knowing When to Intervene: Causal Evaluation of AI Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ayush Sawarni, Wenqian Xing, Tijana Zrnic, Ramesh Johari, Vasilis Syrgkanis

The authors evaluate intervention effects during agent runs by randomizing review or control at eligible decisions. For Qwen3.5 Flash on ImpossibleBench, adversarial feedback after the fifth failed attempt, with one attempt left, increased estimated terminal detected rule violations by 9.4 percentage points, versus 3.3 after the first failure. These are assignment effects in this experiment, not universal predictions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.