Reproduce “Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hoang-Chau Luong, Dat Ba Tran, Lingwei Chen

Researchers from Rochester Institute of Technology and Rowan University developed Diversity-aware RKL (DRKL), a novel distillation objective that enhances knowledge transfer in Large Language Models by mitigating the overconfidence and reduced output diversity observed in standard Reverse Kullback-Leibler (RKL). DRKL consistently achieves higher ROUGE-L scores (up to 3.02 points over RKL) across various benchmarks and student models, while also improving output diversity and prediction calibration.

Key insights

  • RKL's empirical advantage in LLM distillation primarily stems from its mode-seeking behavior, which efficiently focuses learning on high-probability tokens, especially with large vocabularies and capacity mismatch.
  • RKL's overconfidence and reduced diversity are caused by non-target gradients that persistently push the target logit upward, leading to insufficient supervision for non-target (tail) probabilities.
  • Decoupling the target and non-target gradient influences, specifically by removing the (1-q_m) factor, can effectively address RKL's limitations, promoting better non-target distribution alignment and improved output diversity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.