Reproduce “Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ian W. Kennedy, Nafise Sadat Moosavi
Researchers at the University of Sheffield introduce Output-Aware Expectation-Maximisation (OA-EM) to enhance 2-bit quantization of large language models, demonstrating that superior codebook initialisation significantly improves model perplexity and efficiency, even after extensive fine-tuning.
Key insights
- The 'representational ratio' (ρ) between weight groups and codebook capacity predicts the severity of the initialisation bottleneck, with higher ρ indicating greater sensitivity.
- Initial codebook placement fundamentally determines the 'optimization basin' for subsequent search and fine-tuning, meaning initialisation differences persist even after extensive post-quantization fine-tuning.
- At 2 bits per parameter, the limited error correction capacity of subsequent codebooks makes greedy initialisation highly brittle and prone to catastrophic performance degradation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.