Reproduce “PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiva Kumar Pentyala, Zhichao Wang, Bin Bi, Kiran Ramnath, Xiang-Bo Mao, Regunathan Radhakrishnan, Sitaram Asur, Cheng
Salesforce researchers developed PAFT, a parallel training paradigm for LLM fine-tuning that mitigates the 'alignment tax' by concurrently training SFT and preference alignment adapters. By inducing sparsity in SFT adapters via L1 regularization and merging with naturally sparse preference alignment adapters, the method achieves state-of-the-art results, including global Rank #1 on the HuggingFace Open LLM Leaderboard.
Key insights
- Preference alignment methods like DPO naturally induce high sparsity (over 90%) in their delta parameters, while SFT adapters are dense.
- Inducing sparsity in SFT delta parameters through L1 regularization significantly improves the effectiveness of model merging, outperforming explicit sparsification methods.
- The parallel training paradigm, combined with sparse merging, effectively mitigates the 'alignment tax', allowing LLMs to retain SFT capabilities while achieving strong preference alignment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.