Reproduce “Fine-grained Spatiotemporal Grounding on Egocentric Videos” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shuo Liang, Yiwu Zhong, Zi-Yuan Hu, Yeyao Tao, Liwei Wang

Researchers from The Chinese University of Hong Kong created EgoMask, the first pixel-level benchmark for fine-grained spatiotemporal grounding on egocentric videos, alongside a large-scale training dataset, EgoMask-Train. Their work demonstrates that state-of-the-art models, which perform poorly on egocentric data, achieve substantial performance increases—for instance, VideoLISA's IoU_gold pred improved by 41.30%—after fine-tuning on EgoMask-Train, while maintaining or improving performance on exocentric datasets.

Key insights

  • Egocentric videos present inherent challenges like shorter object durations, sparser continuous trajectories, and larger positional shifts, leading to significantly degraded performance for models trained on exocentric data.
  • Fine-tuning models on domain-specific egocentric data (EgoMask-Train) substantially improves performance on egocentric benchmarks while retaining or enhancing performance on existing exocentric datasets, highlighting its complementary value.
  • Traditional metrics like IoU_all are often misleading for egocentric videos with many background frames; alternative metrics such as IoU_gold and IoU_gold pred provide a more accurate assessment of grounding capabilities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.