Reproduce “Pyramid Vector Quantization for LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tycho F. A. van der Ouderaa, Maximilian L. Croci, Agrin Hilmkil, James Hensman

This paper introduces Pyramid Vector Quantization (PVQ) for compressing large language models, leveraging the spherical geometry of LLM weights and an implicit, search-free codebook. The method achieves state-of-the-art compression, notably quantizing Llama-3 70B to 3.25 bits per weight while maintaining approximately 98% accuracy on downstream tasks.

Key insights

  • LLM weights, especially after coherence processing, tend to reside on a hyperspherical manifold, making quantization techniques designed for spherical domains highly suitable.
  • Pyramid Vector Quantization (PVQ) inherently resolves the codebook memory and search complexity issues of traditional vector quantization (VQ) due to its implicit and search-free codebook structure.
  • Normalized amplitudes of LLM weights within groups follow a Beta distribution, enabling optimal, hyperparameter-free amplitude quantization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.