Reproduce “Improving Multi-Modal Learning with Uni-Modal Teachers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, Hang Zhao
The paper defines "Modality Failure" in multi-modal learning, where a weaker modality becomes under-trained in fusion models, and introduces Uni-Modal Teacher (UMT) to address this. UMT achieved 53.46% Top-1 accuracy on VGGSound, surpassing baseline methods, and improved mean Intersection over Union on NYUv2 by up to 1.86%.
Key insights
- "Modality Failure" is a widespread problem in multi-modal learning, where weaker modality encoders are significantly under-trained due to imbalanced contributions and biased joint objectives.
- Theoretical insights from multi-view learning explain how gradients for weaker modalities diminish once stronger modalities adequately reduce the overall loss.
- Distilling knowledge from robust uni-modal teachers into corresponding multi-modal student encoders effectively ensures all modalities receive sufficient training, improving individual representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.