Reproduce “How to post-train on a surrogate: Envelope sampling mitigates reward hacking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sanjit Dandapanthula, Shuvom Sadhuka, Samir Khan, Michael Oberst, Aaditya Ramdas, Alexandra Chouldechova
The authors introduce envelope sampling to recalibrate surrogate rewards before post-training, directing scarce labels toward outputs that a judge-optimized model treats as rare. In a clinical-note experiment, eight labels and 128 updates yielded 0.658 reference quality and 1.0% severe fabrication, versus 0.563 and 19.8% with base-model sampling. The results support this task and protocol, not a general guarantee.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.