Reproduce “ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kai Chen, Ming Dai, Wenxuan Cheng, Wankou Yang

ScanFocus introduces a coarse-to-fine framework for Spatio-Temporal Video Grounding, designed to refine temporal boundaries by decoupling global context understanding from local precision. The system achieves new state-of-the-art results on HC-STVGv1, HC-STVGv2, and VidSTG datasets, demonstrating improvements such as a 5.4% increase in [email protected] on HC-STVGv1 and a 2.6% increase on HC-STVGv2.

Key insights

  • Decoupling the STVG task into coarse global understanding and fine-grained local refinement effectively resolves temporal ambiguity without sacrificing global context.
  • Explicitly modeling inter-frame dependencies within dense local windows, guided by semantic information, is critical for accurately pinpointing temporal start and end boundaries.
  • A deformable attention-based semantic-motion fusion mechanism provides an efficient way to integrate multimodal features, mitigating spatial-temporal noise and reducing computational overhead.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.