Reproduce “ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, Yale Song

Researchers at Seoul National University, NVIDIA Research, and Microsoft Research developed ACAV100M, a dataset of 100 million video clips with high audio-visual correspondence, through an automatic curation pipeline maximizing mutual information. Models pretrained on this large-scale dataset achieve competitive or superior performance on various downstream audio-visual tasks compared to those trained on smaller, manually curated datasets.

Key insights

  • Clustering-based mutual information estimation is more robust to real-world noise in video data compared to NCE-based contrastive methods for identifying strong audio-visual correspondence.
  • Averaging MI estimates across multi-layer feature spaces (both within and across modalities) effectively captures a wider range of abstract concepts and improves correspondence retrieval performance.
  • Scalable approximations like SGD K-means and batch greedy subset selection can maintain high performance while enabling efficient processing of extremely large datasets, overcoming computational bottlenecks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.