Reproduce “The Devil is in Temporal Token: High Quality Video Reasoning Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sitong Gong, Yunzhi Zhuge, Lu Zhang, Zongxin Yang, Pingping Zhang, Huchuan Lu

VRS-HQ, an end-to-end framework, improves video reasoning segmentation by introducing hierarchical temporal tokens and an integrated keyframe selection mechanism. This approach, leveraging SAM2 for mask decoding and propagation, achieved state-of-the-art performance on the ReVOS benchmark, boosting J&F scores by up to 12.5% on reasoning tasks.

Key insights

  • Hierarchical temporal tokens within an MLLM, especially when refined through dynamic aggregation, significantly improve the model's ability to perceive fine-grained spatial details and complex temporal dynamics.
  • Integrating SAM2's occlusion scores with internal token similarity provides a more reliable and data-driven method for keyframe selection, surpassing external models and improving overall accuracy.
  • An end-to-end pipeline that leverages a unified foundation model like SAM2 for both keyframe segmentation and cross-frame mask propagation leads to superior performance and consistency compared to decoupled approaches.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.