Reproduce “AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rameswar Panda, Chun-Fu Chen, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude Oliva, Rogerio Feris
Researchers from the MIT-IBM Watson AI Lab introduced AdaMML, an adaptive multi-modal learning framework for video recognition that dynamically selects modalities for each segment. This approach reduces computational costs by 35-55% while consistently improving or maintaining accuracy compared to fixed multi-modal fusion baselines.
Key insights
- Dynamic, segment-level selection of modalities, rather than fixed multi-modal fusion, can significantly enhance both efficiency and accuracy.
- The Gumbel-Softmax trick enables differentiable, end-to-end training of discrete modality selection policies, simplifying optimization compared to reinforcement learning.
- Incorporating an explicit efficiency loss alongside classification loss is essential for effectively balancing computational cost and recognition performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.