Reproduce “Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jinhao Li, Jiaming Xu, Shiyao Li, Shan Huang, Jun Liu, Yaoxiu Lian, Guohao Dai

This research introduces a framework that achieves near-lossless large language model inference at an average 2.91-bit per weight, delivering up to 1.82x end-to-end speedup and up to 2.88x runtime cost reduction on GPUs for models like Llama2, Llama3, and ChatGLM3. The approach integrates intra-weight mixed-precision, targeted sparse outlier reservation, and asynchronous dequantization to enhance both accuracy and speed.

Key insights

  • Analyzing fine-grained weight group sensitivity allows for adaptive bit-width assignment within a single weight matrix, leading to improved accuracy at very low average bit-widths.
  • Strategically limiting sparse outliers to a minimal ratio and reserving them exclusively for 2-bit groups balances accuracy benefits with minimal computational overhead, keeping speed degradation predominantly within kernel launch times.
  • Identifying and exploiting dequantization dependencies enables asynchronous execution on GPUs, thereby significantly reducing latency by overlapping multiple computational stages.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.