Reproduce “Reasoning Compression with Mixed-Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Yang, Mingyan Wu, Bailan He, Zeyu Cao, Sikuan Yan, Kevin Qinghong Lin, Zifeng Ding
Researchers developed Mixed-Policy Distillation (MPD), a framework enabling smaller large language models to significantly reduce inference-time token usage for reasoning tasks while maintaining or improving accuracy. MPD achieves this by having a larger teacher model compress and refine the student's own generated reasoning trajectories, combining on-policy learning with teacher-guided conciseness.
Key insights
- Larger, more capable LLMs inherently produce shorter, less redundant reasoning traces, making them a natural source of supervision for concise reasoning in smaller models.
- Combining on-policy student trajectory generation with teacher-guided compression effectively mitigates exposure bias while leveraging a stronger model's ability to refine and condense reasoning.
- Reasoning compression can be achieved without explicit length penalties, reward models, or direct ground-truth supervision for conciseness, relying instead on the teacher's qualitative rewriting capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.