Reproduce “Attributing Agent Failures to Their History Needs an Executed Counterfactual: Toward a Reporting Standard” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Egor Pakhomov, Erik Nijkamp

The paper proposes a reporting standard for causal claims about agent history. It requires restoring a mid-run state, changing only the named history, and reporting replay fidelity under a fixed probe. Exact claims require coupled randomness; distributional claims require replicated continuations. In a model-free test, capturing random-stream position reduced divergence from live runs to 0/24, but does not establish behavioral effects or complete state restoration.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.