Reproduce “Variational Information Distillation for Knowledge Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, Zhenwen Dai
Variational Information Distillation (VID) transfers knowledge from a pretrained teacher to a student across architectures. It models teacher activations conditionally on student activations and optimizes a variational lower bound on mutual information alongside task training. On CIFAR-10, a fully connected student with 4,096 hidden units reaches 85.18% test accuracy, versus 76.02% for FitNet. This is evidence for this setting, not deployment efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.