Reproduce “ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Utkarsh Saxena, Sayeh Sharify, Kaushik Roy, Xin Wang
ResQ quantizes language-model weights, activations, and key-value cache by assigning higher precision to a small, high-variance subspace and rotating values within both precision groups. Under a normal-distribution assumption, this reduces an upper bound on quantization error. At nominal 4/4/4, Meta-Llama-3-70B reaches WikiText perplexity 4.1 versus SpinQuant’s 6.2, with some channels retained at 8-bit and queries at 16-bit.
Key insights
- Minimizing activation quantization error is provably achieved by prioritizing high-variance components for higher precision, which can be effectively identified using Principal Component Analysis (PCA).
- Combining outlier suppression via random rotations with a theoretically optimal mixed-precision scheme significantly enhances the effectiveness of low-bit quantization for LLMs.
- Careful design and integration of projection operations into the LLM architecture can preserve the benefits of mixed-precision quantization while incurring minimal runtime overhead during inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.