Reproduce “Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zirui Song, Jiadong Zhang, Zheng Lu, Xiaosong Ma, Xiaojun Chang, Xiuying Chen
SimAudit audits simulated manipulation task checkers by using instructions and checker code to propose candidate states, staging them in simulation, and visually reviewing verdict mismatches. Across 142 tasks, it identified 61 distinct defects: 40 phantom successes and 21 silent failures. On audited RoboDojo tasks, rescoring VPP changed its score from 30.0% to 39.1%; rankings stayed unchanged across three benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.