Reproduce “When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenqiang Luo, Jacky Wai Keung, Boyang Yang, He Ye, Claire Le Goues, Tegawende F. Bissyande, Haoye Tian, Bach Le

This paper empirically investigates federated learning for fine-tuning large language models (LLMs) on private industrial code to perform automated program repair. It demonstrates that federated fine-tuning significantly enhances LLM repair capabilities while preserving data privacy, often outperforming centralized fine-tuning, and reveals unexpected robustness to data heterogeneity.

Key insights

  • Federated fine-tuning not only boosts LLM-based program repair capabilities but can also achieve performance comparable to or even surpass centralized fine-tuning, challenging traditional assumptions about privacy-preserving learning.
  • Large Language Models exhibit unexpected robustness to feature-skewed Non-IID data in federated fine-tuning for program repair, with data heterogeneity often having a negligible or even positive impact on performance.
  • While FedAvg is generally stable, personalized FL algorithms like pFedMe underperform for LLM fine-tuning in this context, indicating a need for new personalization strategies tailored to LLMs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.