Reproduce “Contrastive Audio-Visual Masked Autoencoder” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, James Glass

CAV-MAE combines contrastive audio-visual learning with masked data modeling, matching paired audio and video while reconstructing masked input. Separate modality streams support matching, while a fused stream supports reconstruction. On AudioSet-20K, it reaches 40.5 mAP, versus 38.5 for contrastive-only CAV and 37.4 for reconstruction-only AV-MAE; classification results use labeled fine-tuning.

Key insights

  • Contrastive learning and masked data modeling are complementary paradigms for audio-visual self-supervised learning, and their synergistic combination in CAV-MAE yields superior representations for both classification and retrieval tasks.
  • Multi-modal pretraining can significantly improve single-modal representation learning, with CAV-MAE enhancing audio-only classification performance over models trained with single-modal MAE.
  • The proposed architecture successfully learns versatile representations that are simultaneously 'joint' for multimodal fusion and 'coordinated' for cross-modal matching, addressing a wide range of audio-visual tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.