Reproduce “$\text{Log}_\text{b}$Quant: Quantizing Language Models in Logarithmic Space” on OpenResearch

About this paper

Jeremias Bohn, Tizian Dippold, Mahdi Koubaa, Elias R. Wahl, Georg Groh

Log_b Quant is an adaptive logarithmic quantization method for large language models that maintains model performance at 4-bit precision, unlike linear quantization which causes severe degradation. The approach significantly reduces memory footprint, allowing models up to 14 billion parameters to run on consumer GPUs, and offers moderate inference speedups for small batch sizes.

Key insights

• Adaptive logarithmic quantization effectively preserves LLM performance at aggressive 4-bit precision, a regime where linear quantization causes severe degradation and renders models unusable.
• Energy-based pruning is crucial for 4-bit quantization, as it improves approximation quality by tightening the effective weight range and adapting to different weight distributions.
• Tensor-level, weight-only quantization can yield robust results at low bitwidths, offering substantial memory savings and addressing the memory wall effect for small batch sizes.

Summary by alphaXiv.