Reproduce “SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zikun Qu, Min Zhang, Mingze Kong, Zhiwei Shang, Zhengyu Chen, Yikun Ban, Shuang Qiu, Zhongxiang Dai
Researchers from The Chinese University of Hong Kong, Shenzhen, developed SPOT, a method for on-policy distillation that improves smaller language models' mathematical reasoning by selectively identifying positions for probing and calibrating teacher guidance with verifier-scored outcomes. The approach consistently enhanced solution coverage (Pass@8) by 2.49-5.28 points and average accuracy (Avg@8) by 0.29-1.48 points across various student model scales and benchmarks.
Key insights
- Combining teacher uncertainty, probability mass concentration, and student-teacher mismatch provides a robust signal for identifying high-potential positions for sparse probing.
- Calibrating teacher's local token probabilities with actual downstream outcomes (reward-tilting) is crucial for improving the quality and coverage of learned reasoning paths.
- An explicit, budget-aware strategy for selective probing and outcome calibration can efficiently enhance the student model's ability to explore and exploit diverse correct solutions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.