Reproduce “FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning” on OpenResearch

About this paper

Wang Lu, Xixu Hu, Jindong Wang, Xing Xie

FedCLIP introduces a federated learning framework that leverages a frozen CLIP backbone with a lightweight attention-based adapter (AttAI) to achieve fast generalization and personalization. This approach yields superior performance on unseen and participating clients while reducing trainable parameters by 283 times, enabling efficient deployment of large foundation models in resource-constrained environments.

Key insights

• Freezing the large foundation model backbone while adapting with a tiny, specialized module effectively preserves generalized prior knowledge and prevents performance degradation from limited, non-IID local data.
• An attention mechanism within a lightweight adapter can effectively learn client-specific nuances from pre-extracted visual embeddings, simultaneously enabling strong generalization to unseen domains and personalization for participating clients.
• Significant resource savings can be achieved by federated learning on lightweight adapters (283x fewer parameters) rather than the entire large model, making foundation models feasible in constrained FL environments.

Summary by alphaXiv.