Reproduce “The Asymmetric Harms of LLM Compression” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuan Wu, Mairui Li, Lesia Semenova, Chudi Zhong

This research systematically evaluates the effects of compression on Large Language Models (LLMs), demonstrating that aggregate metrics obscure critical asymmetric harms to knowledge retention, model confidence, and social bias. The study reveals that compressed models can disproportionately degrade rare knowledge, retain high confidence in newly incorrect answers, and shift stereotypical preferences unevenly across demographic subgroups.

Key insights

  • Aggregate performance metrics for compressed LLMs create 'blind spots,' consistently obscuring asymmetric harms to knowledge retention, confidence, and social bias that manifest at granular levels.
  • Compression can disproportionately affect knowledge retention, often showing higher relative retention rates for tail knowledge compared to head knowledge, particularly with pruning methods on specific datasets.
  • Compressed models frequently exhibit overconfidence in newly incorrect answers, especially for knowledge they previously held, and bias shifts tend to concentrate on specific demographic subgroups rather than uniformly affecting overall bias scores.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.