Reproduce “Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jun Luo, Chen Chen, Shandong Wu
A personalized federated learning framework named pFedMoAP enables Vision-Language Models (VLMs) to adapt effectively in decentralized settings by leveraging multiple lightweight prompt experts. It integrates a novel attention-based gating network to dynamically combine local and non-local prompt knowledge, yielding superior accuracy across diverse datasets with significant improvements, such as a 5.94% boost on DomainNet, while also demonstrating robustness under differential privacy.
Key insights
- The lightweight nature of prompts permits sharing multiple pre-aggregated versions between clients via the server, moving beyond the traditional single-global-model restriction in FL.
- An attention-based gating mechanism effectively blends local and collective prompt knowledge, providing a robust and versatile solution for VLM adaptation in federated settings.
- Personalization in federated prompt learning is significantly improved by allowing clients to judiciously select and integrate knowledge from a diverse pool of non-local experts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.