Reproduce “VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zihan Liu, Xinhao Luo, Junxian Guo, Wentao Ni, Yangjie Zhou, Yue Guan, Cong Guo, Weihao Cui and 6 more

VQ-LLM is an automatic framework developed by Shanghai Jiao Tong University researchers that generates high-performance fused kernels for Vector Quantization (VQ)-augmented Large Language Model (LLM) inference. This framework achieves up to 2.2x speedup over unoptimized VQ implementations and delivers comparable or superior latency and accuracy compared to state-of-the-art 4-bit element-wise quantization methods.

Key insights

  • The practical performance limitations of Vector Quantization (VQ) in LLMs stem primarily from inefficient system-level implementations, not inherent algorithmic flaws in VQ.
  • Optimizing codebook access through hierarchical caching, including placing "hot" entries in thread-local registers, is crucial for mitigating shared memory bank conflicts and accelerating dequantization.
  • Reorganizing computation to align with codebook access patterns (codebook-centric dataflow) and fusing dequantization with subsequent computations at the register level significantly reduces on-chip and off-chip memory traffic.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.