Reproduce “One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Lei Liu, Zheng Zhang and 1 more

Researchers at Show Lab, National University of Singapore and Amazon developed VideoLISA, a video-based multimodal large language model capable of language-instructed reasoning segmentation in videos. The model introduces a sparse dense sampling strategy and a 'One-Token-Seg-All' approach to achieve state-of-the-art results on several video object segmentation benchmarks and a new benchmark designed for complex video reasoning, while also excelling at image segmentation tasks.

Key insights

  • Efficiently capturing both high-resolution spatial details and broad temporal dynamics in video for LLMs is crucial for dense prediction tasks.
  • A single, unified token, supervised for multi-frame segmentation, can effectively track and segment objects consistently across video frames by learning a robust spatiotemporal semantic kernel.
  • Joint training on diverse image and video segmentation datasets enhances a model's ability to perform both complex reasoning (from image data) and temporal understanding (from video data).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.