Reproduce “VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, Jiahui Yu

VideoCoCa, developed at Google Research, adapts the image-text CoCa model for diverse video-text tasks by applying its pre-trained attentional poolers to flattened video frame embeddings. This approach enables robust zero-shot transfer and competitive finetuned performance across classification, retrieval, and captioning, requiring minimal video-specific architectural changes or additional training.

Key insights

  • Pre-trained attentional pooling layers from an image-text model are inherently adaptable and highly effective for processing flattened spatial-temporal information from videos without requiring explicit, new temporal modeling modules.
  • Significant performance in diverse video-text tasks can be achieved with minimal architectural changes and zero new learnable parameters by creatively reusing existing components of a strong image-text foundation model.
  • The 'Frozen Encoder Tuning' (LiT) finetuning strategy, which freezes the image encoder but tunes poolers and the text decoder, offers an optimal balance of performance and computational efficiency for adapting to video-specific datasets.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.