Reproduce “TAP-Vid: A Benchmark for Tracking Any Point in a Video” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Carl Doersch, Ankush Gupta, Larisa Markeeva, Adrià Recasens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman and 1 more

DeepMind introduces TAP-Vid, the first comprehensive benchmark and dataset for Tracking Any Point (TAP) in videos, providing precise, long-term trajectories of physical points on surfaces along with explicit occlusion estimation. The work formalizes this problem, offers high-quality real and synthetic data, and establishes a strong baseline model, TAP-Net, which significantly outperforms prior methods.

Key insights

  • The new optical flow-assisted human annotation pipeline significantly boosts speed (28% faster) and precision for generating point tracks, achieving an average inter-rater difference of only 1.46 pixels.
  • TAP-Vid-Kinetics features long, complex trajectories with substantial motion and frequent, long-duration occlusions, demonstrating a high diversity of point behaviors not adequately captured by existing benchmarks.
  • Simple architectural choices in TAP-Net, such as a TSM-ResNet-18 backbone and soft argmax for position prediction, are effective for generalizable point tracking, outperforming adapted state-of-the-art methods like RAFT and COTR by a large margin.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.