Reproduce “The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, Yu Meng

The authors separate reinforcement learning with verifiable rewards into positive sample reinforcement and negative sample reinforcement, then test whether training only on incorrect responses can preserve math reasoning performance across repeated sampling. For Qwen2.5-Math-7B, negative sample reinforcement reaches 53.3% Pass@256 on AIME 2025, above the base model and GRPO, but results vary across model families.

Key insights

  • Negative Sample Reinforcement (NSR) is surprisingly effective, improving Pass@k performance by implicitly preserving prior knowledge and constructively redistributing probability mass, which helps maintain output diversity.
  • Positive Sample Reinforcement (PSR) can lead to diversity collapse by overly concentrating probability on a few correct responses, resulting in degraded Pass@k performance at higher sampling rates.
  • The effectiveness of RL training is highly dependent on the LLM backbone, with Llama-3.1-8B-Instruct showing degradation from RL, while NSR caused the least decline in scaling behavior.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.