Reproduce “Knowledge Distillation with Refined Logits” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wujie Sun, Defang Chen, Siwei Lyu, Genlang Chen, Chun Chen, Can Wang

Researchers from Zhejiang University and the University at Buffalo developed Refined Logit Distillation (RLD), a knowledge distillation method that dynamically refines teacher logits to mitigate misleading information from incorrect predictions while preserving valuable class correlations. RLD achieved superior Top-1 and Top-5 accuracy on the ImageNet dataset and maintained computational efficiency comparable to classical knowledge distillation.

Key insights

  • Dynamically refining teacher logits with label information can effectively mitigate the negative impact of teacher errors without compromising valuable 'dark knowledge' (inter-class correlations).
  • Allowing strategic divergence and greater autonomy for the student model through components like SCD and MCD, rather than strict alignment with all aspects of teacher knowledge, can lead to improved student performance.
  • The combined approach of addressing true-class confidence (SCD) and non-target class correlations (MCD) makes knowledge transfer more robust, particularly when the teacher's predictions are less reliable.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.