Reproduce “Model compression via distillation and quantization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Antonio Polino, Razvan Pascanu, Dan Alistarh

This work introduces Quantized Distillation and Differentiable Quantization, frameworks that effectively combine knowledge distillation with weight quantization to compress deep neural networks. These methods enable significantly smaller models, achieving up to 10x compression and often matching or exceeding the accuracy of larger, full-precision teacher networks across various tasks.

Key insights

  • Jointly leveraging knowledge distillation and weight quantization yields superior compression and accuracy benefits compared to either technique applied in isolation.
  • The 'soft targets' provided by a teacher model during distillation are crucial for guiding the training of low-precision, quantized student models, leading to better performance and convergence.
  • Optimizing the precise locations of quantization points (Differentiable Quantization) and adaptively distributing bits across layers can significantly improve accuracy, especially at very low bit-widths.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.