Reproduce “DistillLens: Symmetric Knowledge Distillation Through Logit Lens” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Manish Dhakal, Uthman Jinadu, Anjila Budathoki, Rajshekhar Sunderraman, Yi Ding
The DISTILLLENS framework facilitates knowledge distillation in Large Language Models by utilizing the Logit Lens to project intermediate hidden states into the vocabulary space. It then applies Jensen-Shannon Divergence to symmetrically align these projected 'evolving belief states' between teacher and student models, resulting in student models that achieve superior performance, reduced exposure bias, and sometimes even exceed teacher model metrics.
Key insights
- Repurposing the Logit Lens from a post-hoc interpretability tool to an active supervision mechanism during training allows for direct, fine-grained alignment of the 'evolving thought processes' within Large Language Models.
- Symmetric divergence objectives, particularly Jensen-Shannon Divergence, provide a theoretically robust mechanism to align intermediate probability distributions, overcoming the limitations of asymmetric KL divergence and MSE by preserving nuanced semantic uncertainty.
- Aligning internal belief states across intermediate layers significantly mitigates exposure bias, enabling student models to achieve greater robustness and generalize better without incurring the substantial computational overhead of on-policy distillation methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.