Reproduce “OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning” on OpenResearch

About this paper

Qinglin Ye, Zhiyuan Gu, Jingjie Xia, Yiheng Zhang, Kaiyan Zhao, Shunchao Zheng, Yuhang Mu, Wenchao Du and 1 more

OPDSearch+ introduces an on-policy distillation framework with reinforcement learning refinement that enables smaller language models to perform search-augmented reasoning for complex question answering tasks. A 3B parameter model trained with OPDSearch+ achieved a mean Exact Match (EM) score of 0.4402 across seven QA benchmarks, surpassing prior 3B reinforcement learning baselines (0.421) and demonstrating particular improvements, such as a 13.1% gain on HotpotQA, for multi-hop tasks.

Key insights

• Forward-KL on-policy distillation with importance ratio clipping offers superior training stability and robustness compared to reverse-KL variants for interactive search environments, effectively providing implicit multi-level supervision and controlling gradient variance.
• A frozen, off-the-shelf instruct model can serve as an effective teacher by reshaping the student's policy distribution, establishing a richer behavioral foundation that enables subsequent RL to converge to superior solutions.
• The two-stage OPD then RL pipeline is crucial for optimal performance, where OPD establishes foundational reasoning and search skills, and RL refines task-specific accuracy; simultaneous optimization proves less effective.

Summary by alphaXiv.