Reproduce “Keypoint-based Progressive Chain-of-Thought Distillation for LLMs” on OpenResearch

About this paper

Kaituo Feng, Changsheng Li, Xiaolu Zhang, Jun Zhou, Ye Yuan, Guoren Wang

This research introduces Keypoint-based Progressive Chain-of-Thought Distillation (KPOD), a framework that improves the transfer of reasoning abilities from large language models to smaller ones by identifying keypoint tokens and scheduling learning progressively. KPOD achieves performance gains over existing CoT distillation methods on arithmetic and logical reasoning benchmarks, enabling smaller models to achieve advanced reasoning with reduced computational cost.

Key insights

• Not all tokens in a Chain-of-Thought rationale are equally important; identifying and emphasizing 'keypoint tokens' can make distillation more efficient.
• Learning reasoning steps in a progressive, easy-to-hard order, similar to human cognition, improves the student model's ability to acquire complex reasoning skills.
• Combining token significance weighting with in-rationale progressive distillation leads to a more effective transfer of reasoning capabilities from large language models to smaller ones.

Summary by alphaXiv.