Reproduce “Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wen-Zhi Li, Sainyam Galhotra, Claire Cardie
TAR selectively replays diverse correct solutions sampled from its base model when training makes them less likely, using rollout- and token-level gates. In a controlled simulation, only 2.6% of runs recovered from under-sampling within 200 updates. Across six math benchmarks, TAR averages 41.2% pass@1 and 61.1% pass@8, versus GRPO’s 37.9% and 60.2%; its bank cannot restore behaviors absent from sampled base-model solutions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.