Reproduce “Trust the uncertain teacher: distilling dark knowledge via calibrated uncertainty” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeonghyun Kim, SooKyung Kim, Richeng Xuan, Hyunsoo Cho
Calibrated Uncertainty Distillation (CUD) introduces a framework to enhance knowledge distillation by shaping the teacher model's predictive distribution to retain meaningful uncertainty and by calibrating the knowledge transfer for selective student imitation. CUD consistently improves student model accuracy, robustness, and out-of-distribution detection capabilities, particularly on tasks with many classes, yielding an average student accuracy of 94.33% for larger student models.
Key insights
- Effective knowledge distillation requires teachers to express difficulty-aware uncertainty, allocating probability mass to semantically related alternatives for challenging inputs rather than overconfidently collapsing to a single class.
- Students should selectively learn from the teacher's informative relational structure, actively suppressing or correcting spurious over-confident peaks that arise from teacher errors.
- The practical modules (DUS and W-Clip) approximate a theoretically grounded constrained optimization problem, enabling efficient and robust transfer of calibrated uncertainty without requiring architectural modifications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.