Reproduce “JudgeRLVR: Judge First, Generate Second for Efficient Reasoning” on OpenResearch

About this paper

Jiangshan Duo, Hanyu Li, Hailin Zhang, Yudong Wang, Sujian Li, Liang Zhao

The JudgeRLVR framework, developed by researchers from Peking University and LLM-Core Xiaomi, introduces a two-stage "judge-then-generate" paradigm to enhance Large Language Model reasoning. This approach improves in-domain math accuracy by an average of 3.7 percentage points while reducing generation length by 42% and demonstrates improved generalization on out-of-domain tasks.

Key insights

• Explicitly training an LLM to "judge" valid reasoning steps and solutions before generative fine-tuning induces a systematic shift in its internal reasoning style, leading to greater efficiency.
• The sequential separation of judging and generating stages is critical, allowing the model to consolidate an error-aware prior that effectively guides subsequent generation, outperforming interleaved or single-stage approaches.
• An internalized discriminative capability enables the model to implicitly prune low-quality reasoning branches, resulting in significantly reduced explicit backtracking and verbosity in generated outputs.

Summary by alphaXiv.