Reproduce “PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Reyhan Kevser Keser, Aydin Ayanzadeh, Omid Abdollahi Aghdam, Caglar Kilcioglu, Behcet Ugur Toreyin, Nazim Kemal Ure
PURSUhInT clusters teacher layer representations by similarity and selects representative center positions as intermediate hints for training a student. This changes hint selection, not the student’s inference architecture. On CIFAR-100, a ShuffleNetV1 student trained from ResNet32x4 with AT+KD reaches 76.98% Top-1 accuracy, versus 75.49% with baseline positions; gains occur in many, but not all, tested settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.