Reproduce “TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstruction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhang Li, Priyadarshini Panda
Yale University researchers introduced TesseraQ, a post-training quantization technique tailored for ultra low-bit (2-bit and 3-bit) quantization of large language models. It achieves superior performance compared to existing methods by employing a block reconstruction approach that optimizes individual weight rounding through Progressive Adaptive Rounding and concurrent dequantization scale tuning, demonstrating significantly lower perplexity on models like LLaMA-2-7B.
Key insights
- Optimizing rounding of individual weight values through a block reconstruction approach is more effective than only distribution transformation for ultra-low bitwidths.
- The Progressive Adaptive Rounding (PAR) technique stabilizes the quantization process by iteratively hardening a subset of rounding variables while optimizing the remaining soft variables.
- Jointly tuning dequantization scale factors during the rounding optimization allows for better adaptation and preservation of the model's functionality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.