Reproduce “Open-Vocabulary Action Localization with Iterative Visual Prompting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
Researchers at Microsoft's Applied Robotics Research developed Temporal PIVOT (T-PIVOT), a training-free framework for open-vocabulary action localization in videos that leverages off-the-shelf Vision-Language Models (VLMs) like GPT-4o. This approach iteratively refines temporal boundaries through visual prompting, achieving state-of-the-art zero-shot performance on the THUMOS14 dataset with an average mAP of 25.8 without any pre-training.
Key insights
- Off-the-shelf VLMs, specifically GPT-4o, can effectively perform complex temporal reasoning and action localization by interpreting spatially arranged frames and iterative visual prompts, without requiring task-specific training.
- Iterative refinement of the temporal sampling window is crucial for converging to precise action start and end times in an open-vocabulary setting.
- The effectiveness of T-PIVOT is highly dependent on the advanced reasoning and prompt interpretation capabilities of the chosen VLM, with GPT-4o significantly outperforming less capable models like Phi-3-vision.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.