Reproduce “Human-centric Spatio-Temporal Video Grounding With Visual Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, Dong Xu
Researchers introduce the Human-centric Spatio-Temporal Video Grounding (HC-STVG) task, a new dataset, and the STGVT model for locating a person's spatio-temporal tube in untrimmed videos based on a language query. The STGVT method achieved an m_vIoU of 18.15% on the HC-STVG dataset and 21.62% on the VidSTG dataset, demonstrating improved accuracy over existing baselines.
Key insights
- The new HC-STVG dataset fills a critical gap, enabling research into combined spatial and temporal grounding for specific human actions in real-world untrimmed videos.
- Visual Transformers effectively integrate visual and textual information, leading to robust cross-modal representations crucial for complex grounding tasks.
- A sequential pipeline of generating tube proposals, matching them with descriptions, and then refining temporal boundaries proves effective for accurate spatio-temporal localization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.