Reproduce “The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Satya Sai Srinath Namburi, Makesh Sreedhar, Srinath Srinivasan, Frederic Sala

A study from the University of Wisconsin - Madison systematically quantified how pruning and quantization affect the parametric knowledge stored in diverse large language model architectures. The research demonstrates that knowledge retention post-compression is highly dependent on the model family and the specific modules targeted, revealing differential sensitivities between attention and feed-forward networks and a critical role for final dense layers.

Key insights

  • The impact of compression on parametric knowledge is highly dependent on the specific model architecture (encoder-only, encoder-decoder, decoder-only).
  • Attention and feed-forward modules exhibit differential sensitivities to pruning versus quantization, often with counter-intuitive results depending on the model type.
  • The final dense layer plays a critical role in encoding parametric knowledge, especially in decoder-only and encoder-decoder models, showing drastic performance drops even at low sparsity levels.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.