Reproduce “GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei Huang, Anda Cheng, Yinggui Wang
GradPruner introduces a fine-tuning approach for Large Language Models that utilizes early-stage gradients to guide layer pruning and a novel symbol-based merging strategy, reducing model parameters by 40% with only a 0.99% average accuracy drop. The method achieves approximately 36% faster training and 39% faster inference, making LLM adaptation more efficient.
Key insights
- Early-stage gradients from LoRA fine-tuning accurately reflect long-term parameter importance for downstream tasks, enabling efficient importance assessment.
- Combining gradient-guided layer pruning with a sign-aware layer merging technique effectively achieves high sparsity while preserving model accuracy.
- The symbol-based merging strategy is crucial for maintaining performance at higher pruning ratios by preventing destructive interference between parameter values.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.