Reproduce “Expanding Language-Image Pretrained Models for General Video Recognition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, Haibin Ling

Researchers developed X-CLIP, a framework for adapting large language-image pretrained models (like CLIP and Florence) to general video recognition. This approach achieves strong performance across fully-supervised, zero-shot, and few-shot settings by incorporating a lightweight cross-frame attention mechanism and a video-specific text prompting scheme, doing so without costly video-text pretraining.

Key insights

  • Existing large-scale language-image models can be effectively adapted for general video recognition without the prohibitive cost of training a video-text model from scratch.
  • A lightweight Cross-frame Communication Transformer (CCT) with alternating cross-frame and intra-frame attention efficiently models spatio-temporal dependencies.
  • A video-specific text prompting scheme can dynamically generate discriminative textual representations, enhancing classification accuracy, particularly for diverse video content.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.