Reproduce “The case for 4-bit precision: k-bit Inference Scaling Laws” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tim Dettmers, Luke Zettlemoyer
This University of Washington research empirically investigates optimal bit-precision for Large Language Model (LLM) inference by introducing bit-level scaling laws. The study identifies 4-bit precision as consistently yielding the best zero-shot accuracy for a given total model memory footprint across various LLMs and scales, with specific data types and small block sizes improving performance.
Key insights
- Bit-level scaling laws indicate that for a fixed total number of model bits, increasing model size while reducing precision to 4-bit generally yields better zero-shot performance.
- For lower precisions (3-bit to 5-bit), small quantization block sizes (e.g., 64-128) and specific data types (quantile or float) improve accuracy, while these factors show diminishing returns at 6-bit and higher.
- Outlier-dependent quantization can stabilize performance at 3-bit, preventing near-random accuracy, but it does not make 3-bit competitive with 4-bit, nor does it benefit 4-bit scaling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.