Reproduce “Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yubo Li, Ramayya Krishnan, Rema Padman
The paired-probe method diagnoses errors in one-pass agent decisions by asking for an action and factual answers independently on the same state. It finds that 73–85% of errors by the six most accurate systems occur when all measured probes are correct. This is behavioral evidence, not proof of internal reasoning; natural probes may omit decision-relevant information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.