Reproduce “Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Run Yang, Runpeng Dai, Jie Sun, Jielei Zhang, Fan Zhou, Hongtu Zhu, Peiyi Li, Longwen Gao
The IDA-OPD method distills a smaller language model from its own sampled text using the teacher’s probability for each sampled token. It estimates whether each update contracts local entropy and softly shrinks selected corrections when teacher and student agree. On AIME 2024, pass@16 is 83.3%, versus 78.7% for OPD and 80.0% for the teacher; results vary by benchmark.
Key insights
- The primary source of cumulative entropy contraction (diversity loss) in sampled-token OPD is not from highly discrepant tokens, but from a large volume of low-discrepancy updates where teacher and student already largely agree.
- The sampled-token advantage (A_y) alone is an insufficient indicator of an update's entropy effect; the student's local probability distribution (D_y) is crucial for determining if an update will increase or decrease policy entropy.
- A targeted, adaptive intervention based on the local entropy influence (I_H(y)) can effectively preserve generation diversity without requiring expensive teacher information or uniformly adjusting entropy, thus maintaining computational efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.