Reproduce “Streamlining Redundant Layers to Compress Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, Hong Chen
LLM-Streamline compresses large language models by pruning redundant layers and replacing their functionality with a lightweight network, also introducing a "stability" metric for robust evaluation. It achieved 93.1% relative accuracy and 83.7% stability on Llama2-7B with 25% pruning, outperforming prior methods while significantly reducing resource requirements.
Key insights
- Transformer-based LLMs exhibit redundant layers where input and output hidden states are highly similar, making them ideal candidates for structural pruning.
- Replacing the functionality of pruned layers with a small, specifically trained lightweight network is more effective and resource-efficient for performance recovery than general fine-tuning of the entire pruned model.
- Evaluating compressed models requires more than just accuracy; a "stability" metric that considers prediction consistency and original model confidence provides a truer assessment of retained capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.