Reproduce “When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Isabel Guan, Dingyuan Zhang, Shipeng Lyu, I-Ming Chen
The authors test a fixed visual judge monitoring a robot placing a cube in a tray. Training uses simulator-confirmed successes; the judge only monitors attempts. Across 20 seeds, success rises from 66.81% to 96.99%, while accepted hovers per attempt fall from 4.97% to 0.86%. Yet conditional acceptance rises from 31.09% to 49.98%, showing coverage shift, not deterioration on identical images in this simulated setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.