Reproduce “Sparse Fine-tuning for Inference Acceleration of Large Language Models” on OpenResearch

About this paper

Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, Dan Alistarh

Researchers from IST Austria and Neural Magic developed SquareHead KD, a knowledge distillation method that enables accurate sparse fine-tuning for Large Language Models. This approach facilitates inducing high levels of weight sparsity during fine-tuning without compromising task accuracy, yielding up to 3.35x CPU speedup for MPT-7B and combining with INT8 quantization for up to 9.08x acceleration.

Key insights

• Distilling knowledge from intermediate feature representations via normalized MSE (SquareHead KD) is crucial for stable and accurate sparse fine-tuning of diverse LLM architectures.
• Weight sparsity can effectively alleviate memory bandwidth bottlenecks in memory-bound LLMs, translating theoretical compression into practical end-to-end inference speedups on both CPUs and GPUs.
• Sparsity and quantization are highly complementary techniques, offering synergistic benefits that lead to significantly greater LLM inference acceleration than either method alone.

Summary by alphaXiv.