Reproduce “`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Waqas Arshid, Mohammad Awrangjeb, Alan Wee-Chung Liew, Yongsheng Gao

CTC² introduces a self-supervised framework for video object segmentation that leverages attention-guided part-level representations and cross-temporal clustering to achieve robust object delineation and tracking. It surpasses previous self-supervised methods, demonstrating G scores of 0.554 on DAVIS-2017 and 0.570 on YouTube-VOS, while maintaining real-time processing speeds of around 35 fps.

Key insights

  • Semantic object parts offer a more stable and robust unit for temporal consistency in video object segmentation compared to granular pixels or entire object representations.
  • The emergent saliency from a frozen Vision Transformer's [CLS] token attention provides a powerful, training-free prior for effectively guiding self-supervised learning towards relevant object regions.
  • Employing saliency-weighted, multi-offset temporal consistency on soft part distributions, combined with regularization, is crucial for achieving high spatial accuracy and temporal coherence.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.