Reproduce “Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pengpeng Qiao, Yang Cao, Lingling Zhang, Guo Cheng, Junwei Chen, Manjiang Yu, Wei Yang Bryan Lim, Masatoshi Yoshikawa

FedAbA addresses heterogeneous federated fine-tuning by aligning client LoRA bases to a previous-round global basis and reconstructing updates before aggregation. This lets clients with different adapter ranks receive rank-sized prefixes. On LLaMA-2-7B, its NLG Avg is 40.90, versus 40.37 for Fed-PLoRA with client ranks from 4 to 64. The default method remains within the warm-started basis’s span and omits reconstruction residuals.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.