Reproduce “TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Finlay G.C. Hudson, James A.D. Gardner, William A.P. Smith
Researchers from the University of York introduced TAPVid-360, a task for persistently tracking any point by its 3D direction from narrow field-of-view videos, even when the point moves out of frame. They developed a scalable ground truth generation method using 360-degree videos and demonstrated that their baseline model, CoTracker360, achieved a 4-fold reduction in angular error for out-of-frame points compared to existing methods.
Key insights
- Leveraging 360-degree videos offers a practical and scalable approach to generate ground truth for persistent 3D directional tracking, circumventing the difficulties of acquiring dynamic 4D scene models.
- Focusing on predicting 3D unit *directions* rather than absolute *positions* or 2D *pixels* enables models to learn allocentric scene representations and maintain object permanence for points far out of frame.
- An existing state-of-the-art 2D tracker can be effectively adapted for 3D directional tracking by modifying its output to predict rotational updates, demonstrating a viable baseline for this new task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.