Reproduce “CPTQuant -- A Novel Mixed Precision Post-Training Quantization Techniques for Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amitash Nanda, Sree Bhargavi Balija, Debashis Sahoo
The CPTQuant framework introduces three mixed-precision post-training quantization techniques for Large Language Models, dynamically allocating bit-widths based on layer sensitivity. It achieved up to 4x compression and a 2x increase in inference efficiency on BERT and OPT models across classification and language modeling tasks, while maintaining minimal accuracy and perplexity drops.
Key insights
- Dynamically assigning precision based on layer sensitivity can significantly reduce LLM resource demands while preserving performance more effectively than uniform quantization.
- The initial and final ~30% of LLM layers tend to be more sensitive to quantization compared to middle layers.
- Different sensitivity metrics and allocation strategies are optimized for specific tasks; for instance, PMPQ performs well for classification, and TDMPQ excels in language modeling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.