Reproduce “Federated Distillation: A Survey” on OpenResearch

About this paper

Lin Li, Jianping Gou, Baosheng Yu, Lan Du, Zhang Yiand Dacheng Tao

This systematic survey reviews Federated Distillation (FD), a framework integrating Federated Learning (FL) and Knowledge Distillation (KD) to address key limitations of traditional FL. It categorizes existing FD schemes based on how they mitigate challenges such as heterogeneity and communication costs, and outlines future research directions.

Key insights

• FD substantially reduces communication overhead by transferring compact knowledge representations (e.g., logits) instead of large model parameters.
• It enables effective collaboration among clients utilizing diverse and heterogeneous model architectures, overcoming a major limitation of traditional FL.
• Sharing distilled knowledge provides an enhanced layer of privacy protection compared to direct sharing of gradients or model parameters.
• FD strategies offer mechanisms to mitigate issues like client-drift and catastrophic forgetting in federated learning environments.

Summary by alphaXiv.