Reproduce “Olica: Efficient Structured Pruning of Large Language Models without Retraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiujun He, Huazhen Lin
Olica is a structured pruning method for Large Language Models that eliminates the need for resource-intensive retraining. It achieves competitive model compression by applying orthogonal neuron decomposition to Multi-Head Attention layers and linear calibration to Feed-Forward Networks, requiring only minimal calibration data and computational resources.
Key insights
- Orthogonal decomposition of combined attention matrices (e.g., WvWoT) allows for preserving the original function while enabling structured pruning of neurons in MHA layers.
- Fast-OND optimizes the SVD process for MHA layers by decomposing individual matrices (Wv) instead of the full product, significantly reducing computational complexity.
- Linear calibration effectively corrects post-pruning errors in FFN layers by modeling residuals with ridge regression, with a Layer Selection Strategy (MC2) identifying the most beneficial layers for calibration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.