Reproduce “AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression” on OpenResearch

About this paper

Atul Kumar Sinha, François Fleuret

Researchers at the University of Geneva and FAIR, Meta, developed AA-SVD, a two-stage anchored and adaptive SVD framework for compressing large language models, which consistently outperforms SVD-based baselines and competes with structured pruning methods. This approach enables more faithful preservation of LLM behavior, reducing the accuracy drop on LLaMA-7B to 8.9% at 0.6 compression ratio with remapping, and allows aggressive compression up to 0.4 where many baselines fail.

Key insights

• Optimizing compression at the transformer block level, rather than solely at individual layers, is a dominant factor for achieving high-quality and robust LLM compression.
• A combined approach that both 'anchors' to the original model's output and 'adapts' to shifted inputs from preceding compressed layers effectively mitigates error accumulation and maintains functional fidelity.
• High-quality LLM compression can be achieved post-training with a relatively small calibration dataset (e.g., 256 samples), making the method efficient and practical for deployment.

Summary by alphaXiv.