Reproduce “Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Orcun Cetintas, Guillem Brasó, Tim Meinhardt, Laura Leal-Taixé
PLANET integrates reconstructed scene geometry into object queries for tracking, while retaining dense recent and sparsely sampled older observations. It uses estimated monocular 3D centers and refines lifted 2D reference points. PLANET achieves 72.6 HOTA on BFT, 2.1 points above the authors’ prior best, and leads HOTA on DanceTrack and SportsMOT; this does not mean it leads every metric.
Key insights
- Explicitly embedding estimated 3D scene information directly into the model's query representations, rather than simple feature injection, is crucial for improving both detection and association accuracy in MOT.
- Leveraging and refining existing 2D localization priors through an 'anchored' approach is more effective for integrating 3D localization into a detector than attempting to learn independent 3D reference points.
- Extended temporal memory mechanisms (DRTM) are synergistically more effective for identity reasoning when combined with 3D-aware (world-grounded) queries, as stable 3D spatial evidence enhances long-term association.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.