Reproduce “End-to-End Learning of Visual Representations from Uncurated Instructional Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, Andrew Zisserman
Researchers developed MIL-NCE, a self-supervised approach that learns generic visual representations directly from over 100 million uncurated instructional videos and their noisy narrations. This method achieves state-of-the-art performance across various video understanding tasks, often outperforming models pre-trained on manually annotated datasets.
Key insights
- The Multiple Instance Learning (MIL) component, which considers a 'bag' of temporally nearest narrations as potential positives, is essential for robustly handling the inherent temporal misalignment between video and noisy speech transcripts.
- A simple language model based on fixed word2vec embeddings effectively captures semantic information from noisy ASR transcripts, outperforming more complex models like LSTMs or Transformers.
- Symmetric negative sampling within the Noise Contrastive Estimation (NCE) framework significantly enhances the learning of joint video-text embeddings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.