Reproduce “Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models” on OpenResearch

About this paper

Xinxin Liu, Aaron Thomas, Cheng Zhang, Jianyi Cheng, Yiren Zhao, Xitong Gao

The research systematically evaluates sparsity-based Parameter-Efficient Fine-Tuning (SPEFT) for Large Language Models, demonstrating that a simple gradient-based, static masking strategy consistently outperforms more complex SPEFT methods and often surpasses low-rank adaptation techniques like LoRA, particularly on reasoning and generation tasks. This approach achieves high performance with negligible computational and memory overhead, making LLM fine-tuning more resource-efficient.

Key insights

• Simple gradient-based salience metrics are more effective and robust for determining sparsity masks in SPEFT than complex second-order or data-free metrics.
• Static sparsity masks, determined once before training, are sufficient and more computationally efficient than dynamic masks, which offer only marginal or no performance benefits for larger LLMs.
• SPEFT can match or exceed the performance of leading low-rank PEFT methods like LoRA and PiSSA, especially on complex reasoning and generation tasks, with comparable parameter budgets and negligible overhead.

Summary by alphaXiv.