Reproduce “Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaobing Chen, Zhiqi Pang
Residual Advantage (RA) adds token-level teacher guidance to verifier-based reinforcement learning for math reasoning. At each sampled step, it compares teacher and student probabilities, subtracts a student-policy baseline, then centers the guidance within the response so the verifier’s mean label is unchanged. On Qwen3-1.7B-Base with REINFORCE++, three-seed macro Avg@8 and Pass@8 rose from 25.67% and 39.44% to 29.25% and 45.20%, respectively, across three math benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.