Reproduce “POPri: Private Federated Learning using Preference-Optimized Synthetic Data” on OpenResearch

About this paper

Charlie Hou, Mei-Yu Wang, Yige Zhu, Daniel Lazar, Giulia Fanti

The POPri algorithm enables private federated learning by reinterpreting client feedback as a reinforcement learning reward to fine-tune a Large Language Model (LLM) for differentially private synthetic data generation. This approach closes the performance gap between fully-private and non-private settings by up to 58%, outperforming existing synthetic data and DP-FL methods, while introducing the LargeFedBench benchmark for robust LLM evaluations.

Key insights

• Reframing client feedback as a reinforcement learning reward for LLM policy optimization (specifically DPO) is more effective for generating high-quality differentially private synthetic data than iterative prompt engineering or simple sample selection.
• Dense, privacy-preserved preference scores (e.g., average cosine similarity) from clients are crucial for constructing meaningful 'chosen' vs. 'rejected' pairs for DPO, offering a substantial advantage over sparse feedback mechanisms like nearest-neighbor histograms.
• The optimal selection of a 'rejected' synthetic sample, such as the 5th-ranked out of 10, is critical for DPO effectiveness; samples that are too dissimilar or too similar can render preference feedback uninformative or noisy.

Summary by alphaXiv.