Reproduce “DELTA: Dense Efficient Long-range 3D Tracking for any video” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tuan Duc Ngo, Peiye Zhuang, Chuang Gan, Evangelos Kalogerakis, Sergey Tulyakov, Hsin-Ying Lee, Chaoyang Wang

DELTA, developed by UMass Amherst and Snap Inc., presents a feed-forward neural network for dense, long-range 3D tracking of every pixel from monocular video. The system achieves improved accuracy and efficiency, demonstrating an EPE of 3.63 on the CVO-Extended dataset, outperforming DOT's 4.98, and an AJ of 81.4 on Kubric-3D, significantly better than DOT-3D's 72.3.

Key insights

  • Efficient dense long-range 3D tracking in a feed-forward network is achievable by decoupling global and local spatial attention and processing information at multiple resolutions.
  • Log-depth representation is empirically superior for 3D tracking, improving accuracy due to its enhanced precision for closer objects and its inherent scale-invariance to monocular depth estimations.
  • An attention-based upsampler with static spatial bias effectively reconstructs high-resolution tracks from coarse predictions, outperforming traditional CNN-based or non-learnable upsampling operators.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.