Reproduce “Fluctuation-based Adaptive Structured Pruning for Large Language Models” on OpenResearch

About this paper

Yongqi An, Xu Zhao, Tao Yu, Ming Tang, Jinqiao Wang

FLAP introduces a retraining-free structured pruning framework for large language models, leveraging a fluctuation metric and bias compensation to achieve substantial parameter reduction and inference speedup while maintaining strong performance on language modeling and zero-shot tasks. This approach enables more efficient LLM deployment on general-purpose hardware by directly reducing model size.

Key insights

• Certain channels within LLM hidden states exhibit 'structured sample stability,' meaning their values are consistent across different samples, which can be leveraged for effective pruning and compensation.
• A simple, retraining-free bias compensation mechanism can largely recover performance loss from structured pruning, demonstrating a powerful alternative to expensive fine-tuning methods.
• Adaptive global pruning, guided by standardized importance metrics, is crucial for preserving LLM performance, as different architectural components have varying sensitivities to pruning.

Summary by alphaXiv.