Reproduce “Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zirui Song, Jiadong Zhang, Zheng Lu, Xiaosong Ma, Xiaojun Chang, Xiuying Chen

SimAudit audits simulated manipulation task checkers by using instructions and checker code to propose candidate states, staging them in simulation, and visually reviewing verdict mismatches. Across 142 tasks, it identified 61 distinct defects: 40 phantom successes and 21 silent failures. On audited RoboDojo tasks, rescoring VPP changed its score from 30.0% to 39.1%; rankings stayed unchanged across three benchmarks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.