Reproduce “Selective Aggregation for Low-Rank Adaptation in Federated Learning” on OpenResearch

About this paper

Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, Liangqiong Qu

Researchers at the University of Hong Kong, Stanford, and the Chinese Academy of Sciences developed FedSA-LoRA, an approach for federated fine-tuning of large language models that improves performance and communication efficiency by selectively aggregating LoRA's general knowledge matrices (A) while keeping client-specific matrices (B) local. The method demonstrates superior accuracy and faster convergence across various NLU and NLG tasks, particularly under data heterogeneity.

Key insights

• LoRA's A matrices primarily learn general, transferable knowledge, while B matrices are essential for capturing client-specific knowledge in a federated setting.
• Leveraging this asymmetry by selectively aggregating A matrices enables personalized model updates for each client while maintaining global consistency.
• The selective aggregation strategy (FedSA) provides a general paradigm for effectively integrating various LoRA-based parameter-efficient fine-tuning methods into Federated Learning.

Summary by alphaXiv.