Reproduce “Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Christian Moya, Elliott Thornley, Guang Lin
The authors analyze how RLVR can increase verifier acceptance while reducing correctness when accepted errors are possible. They prove verifier-only records cannot identify or correct this dynamic and propose PAC, which uses audits while preserving acceptance growth. In a Qwen2-0.5B N40 experiment, PAC with 25% audits reached 95.2% correctness versus 2.2% for GRPO; this is not a general guarantee.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.