Reproduce “Deep Mutual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ying Zhang, Tao Xiang, Timothy M. Hospedales, Huchuan Lu
Deep Mutual Learning (DML) introduces a collaborative training framework where multiple neural networks simultaneously learn from both ground truth data and each other's predictions, bypassing the need for a pre-trained teacher network. This method consistently improves model performance, achieving a mean Average Precision of approximately 56% with five networks compared to 47% for independently trained models, and often surpasses traditional knowledge distillation results.
Key insights
- Deep Mutual Learning leads to more robust solutions by encouraging networks to find wider minima in the loss landscape, associated with better generalization.
- DML produces more informative probability distributions with lower certainty, assigning non-negligible probability mass to secondary classes, unlike the overconfident predictions of independently trained models.
- Learning from diverse individual peer perspectives is more beneficial than learning from a single ensemble perspective, as the original DML outperformed the DML with Ensemble Teacher (DML_e).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.