Reproduce “Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Baolong Bi, Shenghua Liu, Yiwei Wang, Siqian Tong, Lingrui Mei, Yuyao Ge, Yilong Xu, Jiafeng Guo and 1 more
The RGR-GRPO framework, developed by researchers from the Chinese Academy of Sciences and the University of California, Merced, enhances multi-domain reasoning in Large Language Models by integrating rubric-based fine-grained rewards with controlled off-policy exploration. This approach consistently outperforms existing reinforcement learning methods, achieving an average improvement of 25.1% over base models and 6.7% over official Instruct models across various reasoning benchmarks.
Key insights
- Fine-grained, rubric-based rewards, comprising both factual and process criteria, provide reliable and dense supervision for LLMs, effectively overcoming the limitations of sparse, domain-limited reward signals in multi-domain reasoning.
- A controlled, adaptive off-policy exploration mechanism, guided by rubrics and an Exploration Assessment, can stably expand the policy's reachable reasoning space, enabling LLMs to discover a broader range of high-quality solutions without causing training instability.
- The combination of structured reward signals with adaptive offline guidance is crucial for fostering robust and generalizable reasoning skills, allowing LLMs to effectively tackle complex, interdisciplinary problems and generalize to out-of-distribution tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.