Reproduce “TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jisu Nam, Jahyeok Koo, Soowon Son, Jaewoo Jung, Honggyu An, Junhwa Hur, Seungryong Kim

TrackCraft3R adapts a video diffusion transformer to track the same first-frame 3D points through monocular video, using external per-frame depth and camera estimates, reference anchoring, and temporal alignment. It predicts trajectories and visibility in one clip pass. With DA3 geometry, it reaches 0.6785 average Jaccard versus 0.4975 for DELTAv2, under the paper’s five-benchmark protocol.

Key insights

  • Pre-trained spatio-temporal priors from video DiTs are crucial for robust dense 3D tracking, significantly outperforming models initialized randomly.
  • The dual-latent representation and temporal RoPE alignment effectively bridge the architectural gap between frame-anchored generative models and the requirements of reference-anchored 3D tracking.
  • Predicting residual 3D displacements from a reference frame, rather than absolute positions, stabilizes training and enhances accuracy by focusing on motion-induced changes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.