Reproduce “Format Drift Masquerades as Reward Hacking: A Pre-Registered Study of When RL Exploits Flawed Checkers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Himanshu Sangshetti
The paper tests whether reinforcement learning amplifies existing checker exploits or creates new ones, and how reliably those exploits are measured. The authors train Qwen3-0.6B for 120 steps and find that a revised, post-hoc math referee removes most apparent exploitation; under a number-anywhere checker, one of two seeds still shows a 42-percentage-point rise in paid-but-wrong answers. Larger-scale discovery remains unresolved.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.