Reproduce “FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients” on OpenResearch

About this paper

DaiXun Li, Weiying Xie, ZiXuan Wang, YiBing Lu, Yunsong Li, Leyuan Fang

A multi-modal collaborative diffusion federated learning framework, FedDiff, enables secure and efficient land cover classification from distributed remote sensing data by integrating diffusion models for robust feature extraction within a lightweight federated structure, consistently outperforming existing methods and reducing communication overhead.

Key insights

• Diffusion models, when adapted with components like Multi-Head Self-Attention and Frequency Domain Parsers, can effectively extract robust multi-modal features that capture spectral, spatial, and frequency domain information.
• Integrating diffusion models within a federated learning framework enables privacy-preserving collaborative analysis of multi-modal data from distributed clients, a previously unexplored combination.
• Communication efficiency in federated multi-modal learning can be significantly improved by leveraging low-rank singular value decomposition to compress high-dimensional feature maps without compromising classification performance.

Summary by alphaXiv.