Reproduce “VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xiaoying Tang, Dianbo Sui, Qingbin Liu, Xi Chen and 1 more
VTG-LLM enhances video Large Language Models by integrating timestamp knowledge for improved Video Temporal Grounding. The model outperforms existing 7B video LLMs in zero-shot settings across dense video captioning, moment retrieval, and highlight detection tasks, demonstrating advanced temporal localization accuracy.
Key insights
- Explicitly embedding absolute timestamp information within visual tokens and using dedicated "absolute-time tokens" for output are vital for video LLMs to perform precise temporal grounding.
- Slot-based token compression offers an efficient method to handle extended video sequences and a higher number of frames within LLM context limits, outperforming naive sampling or general cross-attention.
- A meticulously re-annotated and task-balanced instruction tuning dataset significantly enhances a video LLM's ability to learn and generalize across various temporal grounding tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.