Reproduce “Auditing Reward Hackability in Code RL Training Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shreshth Rajan
This research quantifies "reward hackability" in code reinforcement learning benchmarks, finding 28.5% of SWE-bench Verified tasks and 25.0% of R2E-Gym tasks are vulnerable. It also introduces a sanity-gated repair loop that effectively identifies and filters a 61.9% defect rate in LLM-generated test augmentations, providing a method to harden test suites.
Key insights
- A significant fraction of tasks in major code RL benchmarks are reward-hackable, implying models can achieve high reported performance without genuinely solving the underlying problems.
- Reward hackability demonstrably inflates model Pass@1 scores, with models performing substantially better on hackable tasks compared to robust ones, even when controlling for human-rated difficulty.
- LLM-generated test augmentations exhibit a high intrinsic defect rate (over 60%), underscoring the critical need for execution-grounded sanity checks like the Docker gold-sanity gate to ensure test validity during test suite hardening.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.