Reproduce “Teacher-Guided One-Shot Pruning via Context-Aware Knowledge Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md. Samiul Alim, Sharjil Khan, Amrijit Biswas, Fuad Rahman, Shafin Rahman, Nabeel Mohammed
A teacher-guided one-shot unstructured pruning framework integrates knowledge distillation (KD) directly into the parameter importance score estimation. This approach achieves substantial computational efficiency, pruning models like ResNet18 by 95% on CIFAR-100 with 74.68% accuracy, while being up to 41 times faster than iterative pruning techniques.
Key insights
- Directly integrating knowledge distillation into the parameter importance score calculation process enables more effective pruning masks by leveraging the teacher's knowledge.
- A robust measure of parameter importance can be derived from stable gradient signals that combine both task-specific loss and a sophisticated context-aware knowledge distillation loss, smoothed using Exponential Moving Average.
- Employing a one-shot global pruning strategy significantly reduces the computational overhead typically associated with iterative train-prune-retrain methods while maintaining high performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.