Reproduce “Video and Audio are Images: A Cross-Modal Mixer for Original Data on Video-Audio Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zichen Yuan, Qi Shen, Bingyi Zheng, Yuting Liu, Linying Jiang, Guibing Guo
Researchers at Northeastern University developed CMMixer, a framework that processes video frames and audio spectrograms as images within a unified masked autoencoder pre-training, achieving up to 2x improvement in video-audio retrieval on M2M and YouTube-8M datasets. The model learns robust, modality-common representations that also transfer effectively to action and audio classification tasks.
Key insights
- Treating audio spectrograms as 'images' enables a unified masked modeling pre-training approach, effectively bridging the representational gap between video and audio to learn robust modality-common features.
- A single shared decoder (CMMixer-Solo) is more effective for fostering modality-common representations during reconstruction compared to using independent decoders for each modality.
- The CutMix mixing method, which creates clear spatial boundaries during fusion, consistently outperforms Pixel Mixup and global Mixup for multimodal pre-training, leading to improved semantic modeling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.