Reproduce “Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data” on OpenResearch

About this paper

Honglu Zhou, Xiangyu Peng, Shrikant Kendre, Michael S. Ryoo, Silvio Savarese, Caiming Xiong, Juan Carlos Niebles

Strefer introduces an automatic data engine that generates synthetic, fine-grained, spatiotemporally rich instruction data, enabling Video LLMs to better understand and reason about specific spatial regions and temporal moments within videos. This approach significantly enhances their performance on a range of spatiotemporal tasks, addressing the scarcity of high-quality training data.

Key insights

• The quality and specificity of video instruction-tuning data, rather than mere quantity, are paramount for effectively training Video LLMs in fine-grained spatiotemporal reasoning.
• Carefully designed traditional instruction data, especially those focusing on dynamic entity behaviors and including negative questions, can substantially improve performance across various spatiotemporal understanding tasks.
• There can be a trade-off between enhancing higher-level, long-term temporal reasoning and maintaining precision in fine-grained, localized spatiotemporal comprehension within current model architectures.

Summary by alphaXiv.