Reproduce “An Empirical Study of Federated Prompt Learning for Vision Language Model” on OpenResearch

About this paper

Zhihao Wang, Wenke Huang, Tian Chen, Zekun Shi, Guancheng Wan, Yu Qiao, Bin Yang, Jian Wang and 2 more

An empirical study systematically investigates Federated Prompt Learning (FPL) for Vision Language Models (VLMs) by analyzing the distinct behaviors of language and vision prompt tuning under varying label skew and domain shift conditions in federated environments. The research reveals that Vision Prompt Tuning outperforms Language Prompt Tuning in label skew scenarios, while Language Prompt Tuning shows better adaptation to domain shifts, and a combined approach achieves superior performance in complex heterogeneous settings.

Key insights

• Vision Prompt Tuning (VPT) consistently outperforms Language Prompt Tuning (LPT) in label skew scenarios due to its more consistent optimization directions, effectively aligning visual features.
• Language Prompt Tuning (LPT) generally performs better in domain shift scenarios, as its less consistent optimization directions allow better adaptation to domain-specific textual characteristics.
• Federated Prompt Learning (FPL) demonstrates robustness across varied federated settings, including communication epochs, aggregation strategies, client scales, and prompt lengths, consistently outperforming zero-shot CLIP.

Summary by alphaXiv.