Reproduce “Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanoona Rasheed, Haania Siddiqui, Ming-Hsuan Yang, Fahad Shahbaz Khan, Salman Khan
The authors introduce Parallel Tube Decoding (PTD) for spatio-temporal video grounding, first predicting an event interval and jointly generating a bounding box for each time index. Decoupled Block Attention removes dependence between boxes while retaining video-query context. On VidSTG, PTD reaches 0.4 seconds TCL, 45.9 BPS, and 32.9 m_vIoU, versus 31.6, 0.5, and 27.4 for unquantized decoding in the setup.
Key insights
- Eliminating trajectory-level dependencies by generating all spatial localizations in parallel significantly reduces decoding latency and prevents error propagation, leading to more efficient and accurate spatio-temporal grounding.
- Decoupled block attention ensures that each spatial prediction remains directly grounded in relevant visual evidence from the video, mitigating the issue of attention shifting to previously generated text in autoregressive models.
- A unified MLLM generative interface can achieve highly efficient and dense spatio-temporal localization without requiring external, specialized spatial prediction modules, simplifying the architecture.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.