Reproduce “Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling” on OpenResearch

About this paper

Yanzhao Hou, Jiaxiang Geng, Boyu Li, Xiaofeng Tao, Juncheng Wang, Xiaodong Xu, Bing Luo

Researchers from Beijing University of Posts and Telecommunications and collaborators developed an adaptive federated Low-Rank Adaptation (LoRA) strategy to fine-tune Large Language Models (LLMs) in heterogeneous wireless networks. Their method jointly optimizes client sampling and LoRA parameter usage, achieving up to 4.2x faster convergence in wall-clock time compared to prior methods on commonsense reasoning tasks.

Key insights

• A new convergence bound for Federated LoRA was derived, accounting for arbitrary sketching ratios and independent client sampling without assuming bounded gradients.
• The analysis demonstrated a complex interdependence between client sampling probabilities and LoRA parameter sizes; their joint optimization is essential for practical efficiency in heterogeneous settings.
• Optimizing for wall-clock time requires a holistic approach that considers per-round latency (influenced by slowest participating clients and communication/computation costs) alongside the number of training rounds.

Summary by alphaXiv.