Reproduce “EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang and 3 more

Researchers from Shanghai AI Laboratory and Nanjing University developed EgoVideo, an egocentric foundation model leveraging a 3-stage training paradigm to adapt a general video-language model for first-person video understanding. The model achieves state-of-the-art performance across eight EgoVis CVPR 2024 Challenge tasks, including natural language queries, action anticipation, and multi-instance retrieval, demonstrating its ability to bridge the domain gap.

Key insights

  • A targeted multi-stage training approach effectively transfers knowledge from general video foundation models to the distinct characteristics and demands of egocentric video.
  • Systematically adapting a powerful pre-trained video-language model with a large, curated egocentric dataset yields highly specialized and robust first-person video representations.
  • The developed EgoVideo model demonstrates strong generalization capabilities, performing well even in domain adaptation settings where it is fine-tuned only on the source domain.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.