Reproduce “Slot-VLM: SlowFast Slots for Video-Language Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, Yan Lu

Slot-VLM introduces a SlowFast Slots module that generates semantically decoupled, fixed-length object-centric and event-centric tokens from videos, improving video question-answering performance. The framework achieves state-of-the-art results on MSVD-QA, MSRVTT-QA, and ActivityNet-QA by providing more interpretable visual context to Large Language Models.

Key insights

  • Semantically decoupled visual tokens, representing distinct objects or events, are crucial for effective video-language alignment and reasoning with LLMs, mirroring how LLMs process discrete linguistic concepts.
  • A dual-branch SlowFast design, separating object-centric and event-centric processing, provides a more comprehensive and complementary visual context for video understanding.
  • Slot Attention-based mechanisms are more effective than Q-Formers for generating truly semantically distinct and interpretable visual tokens, avoiding the entanglement issues of prior aggregation methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.