Reproduce “LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruihao Gong, Yang Yong, Shiqiao Gu, Yushi Huang, Chengtao Lv, Yunchen Zhang, Xianglong Liu, Dacheng Tao
Researchers from Beihang University, SenseTime Research, and Nanyang Technological University introduced LLMC, a versatile toolkit and standardized benchmark for large language model quantization. This work systematically investigates the impact of calibration data, various algorithms, and data formats, revealing that while FP quantization generally excels for weight-activation, INT often performs better for ultra-low-bit weight-only quantization, and enables compressing models up to 175B parameters on a single 40GB A100 GPU.
Key insights
- Calibration data quality, including consistent token distribution and preserved intra-sentence logic, significantly impacts quantization accuracy, especially for learning-based methods.
- Scaling-based transformations can reduce activation outliers but may increase weight outliers, while rotation-based transformations synergize well with reconstruction methods like GPTQ by reducing weight outliers.
- Floating-point (FP) quantization consistently outperforms integer (INT) quantization for weight-activation (e.g., w4a4), but INT can be superior for ultra-low-bit (less than or equal to 3-bit) weight-only quantization due to FP format limitations and group range characteristics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.