Reproduce “Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruihan Jin, Pengpeng Shao, Zhengqi Wen, Jinyang Wu, Mingkuan Feng, Shuo Yang, Chu Yuan Zhang, Jianhua Tao
Knowledge Purification was introduced for multi-teacher knowledge distillation in LLMs, aimed at consolidating or selecting teacher rationales to mitigate knowledge conflicts and improve distillation efficiency. Router-based methods, especially the similarity-based router and RL-based selection, demonstrated up to 6.9% higher average accuracy and reduced GPU hours compared to traditional baselines.
Key insights
- Knowledge conflicts among multiple teacher LLMs are a significant impediment to effective distillation and can worsen as more teachers are added, necessitating proactive purification.
- Router-based knowledge purification methods are highly effective in mitigating these conflicts and improving student model accuracy, offering superior adaptability to increasing teacher ensembles.
- Trained LLM routers can achieve both substantial performance gains and significant efficiency improvements, reducing computational costs and latency while demonstrating strong generalization capabilities on out-of-domain datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.