Reproduce “Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Yang, Hongpeng Pan, Qing-Yuan Jiang, Yi Xu, Jinghui Tang
Researchers developed Adaptively Masking Subnetworks Considering Modal Significance (AMSS and AMSS+), a method that rebalances multi-modal learning by performing fine-grained, element-wise subnetwork optimization. AMSS+ consistently surpasses existing methods across diverse multi-modal tasks, for example, achieving a 5.15% accuracy improvement over the second-best approach on the Kinetics-Sound dataset.
Key insights
- Fine-grained, element-wise modulation of parameters is more effective for rebalancing multi-modal learning than coarse-grained, global-wise modulation.
- Adaptively masking subnetworks based on a combination of modal significance (mutual information rate) and parameter importance (Fisher information) can effectively alleviate modality suppression.
- Providing an unbiased stochastic gradient estimator, as in AMSS+, improves the convergence rate and overall performance of subnetwork optimization in this context.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.