Reproduce “QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models” on OpenResearch

About this paper

Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, Dan Alistarh

QUIK enables practical and accurate end-to-end 4-bit inference for generative Large Language Models by quantizing both weights and activations. This method achieves up to a 3.4x throughput increase and a 3x memory reduction for models like LLaMA2-70B and Falcon-180B with minimal accuracy loss.

Key insights

• A small, strategically chosen percentage of 'outlier' features in activations and their corresponding weights, kept in higher precision, is crucial for achieving accurate 4-bit quantization of the vast majority of an LLM.
• Key LLM inference scenarios, such as prompt processing and batched inference, are compute-bound, making native low-precision (INT4) computation essential for substantial end-to-end speedups.
• Effective algorithm-hardware co-design, including fused CUDA kernels and integrating dequantization into the matrix multiplication epilogue, is vital to translate algorithmic gains into practical throughput improvements by mitigating overheads.

Summary by alphaXiv.