Reproduce “Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, Cordelia Schmid

The Vid2Seq framework introduces a unified visual language model that generates both temporal event boundaries and natural language captions for untrimmed videos as a single sequence. Through large-scale pretraining on narrated video transcripts and fine-tuning, it achieves state-of-the-art performance across dense video captioning, video paragraph captioning, and video clip captioning benchmarks, notably improving YouCook2 CIDEr by 18.2 points.

Key insights

  • Unified sequence-to-sequence modeling, leveraging special 'time tokens,' effectively integrates temporal localization and textual description within a single language model framework.
  • Large-scale pretraining on weakly supervised untrimmed narrated videos provides a scalable solution to data scarcity, enabling the model to learn robust representations from noisy speech transcripts.
  • The combination of generative and denoising pretraining objectives fosters strong multi-modal reasoning and aligns visual and linguistic cues for fine-grained video understanding tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.