Reproduce “Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuyi Yu, Yaohua Liu, Chengjun Li, Qiang Tang, Shuzhe Tang, Kuizhi Mei
Researchers from Xi'an Jiaotong University developed Shift-Augmented Knowledge Distillation (SAKD), a framework that enhances knowledge transfer by generating diverse, adaptive virtual teacher views through student-guided perturbations and parameter-free cyclic shifts. SAKD improves student model accuracy and generalization while simplifying the training process, outperforming prior single-teacher augmentation methods in efficiency and performance.
Key insights
- Leveraging the student's evolving features to dynamically condition perturbation generation enables single-stage training and eliminates the need for a separate pre-training phase for view generators.
- By focusing on perturbing the teacher's semantically correct output rather than reconstructing full logit distributions, the generator's task is simplified, and stability is maintained.
- Employing parameter-free cyclic shifts efficiently creates multiple diverse virtual views from a single base perturbation, ensuring constant parameter complexity and scalability.
- The approach balances the semantic correctness of the original teacher with complementary diversity from student-guided perturbations, leading to more robust student learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.