Reproduce “Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Uygar Kurt
Uygar Kurt conducted a unified empirical evaluation of 13 `llama.cpp` GGUF quantization formats on the Llama-3.1-8B-Instruct model, providing a practical guide on quality, memory footprint, and inference speed trade-offs. The study found that 5-bit Q5_0 quantization reduced model size by 65.19% while slightly improving aggregate downstream task performance over the FP16 baseline, and Q4_K_S offered a robust 70.83% size reduction with minimal quality loss.
Key insights
- The relationship between nominal bit-width and model quality is not strictly monotonic; specific quantization format implementations (e.g., K-quants) significantly influence performance outcomes.
- Mid-bit quantization, particularly 5-bit schemes like Q5_0, can achieve substantial memory reduction (approx. 65%) while maintaining or even slightly improving aggregate downstream task performance compared to the FP16 baseline.
- Task sensitivity to quantization varies considerably, with mathematical reasoning (GSM8K) being highly sensitive, while commonsense reasoning (HellaSwag) remains relatively stable.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.