Reproduce “Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xu Guo, Qiming Ge, Jian Tong, Kedi Chen, Jin Zhang, Xiaogui Yang, Xuan Gao, Haijun Lv and 3 more
Researchers from Shanghai AI Laboratory and Fudan University introduce Iterative Distractor Curation (IDC), a framework that optimizes multiple-choice question (MCQ) design to enhance Reinforcement Learning with Verifiable Rewards (RLVR). This approach improves LLM accuracy by over 3% on medical and law benchmarks and significantly reduces spurious reward rates by leveraging adaptively generated, high-quality distractors.
Key insights
- RLVR performance is highly sensitive to the consistency between training and testing option counts, demonstrating an "option-count distribution bias" where performance degrades with increasing mismatch.
- Distractor strength is a critical factor for effective RLVR, as training with strong, model-based distractors forces deeper reasoning, while weak distractors lead to superficial learning and reward hacking.
- LLMs can effectively self-generate high-quality, adversarial distractors tailored to their own internal error patterns, enabling an autonomous self-improvement loop for RLVR training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.