Reproduce “CRISP: Contrastive Residual Injection and Semantic Prompting for Continual Video Instance Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Baichen Liu, Qi Lyu, Xudong Wang, Jiahua Dong, Lianqing Liu, Zhi Han

The CRISP framework addresses Continual Video Instance Segmentation by enabling models to incrementally learn new object categories while mitigating catastrophic forgetting and maintaining temporal consistency. It achieves state-of-the-art performance on YouTube-VIS datasets, demonstrating superior object tracking and segmentation with significantly reduced forgetting rates.

Key insights

  • Catastrophic forgetting in CVIS manifests as specific instance-wise, category-wise, and task-wise semantic confusions, requiring targeted architectural and loss-based interventions.
  • Injecting residual semantic prompts derived from a text encoder into the Transformer decoder helps align object queries with class-specific knowledge, improving category distinction.
  • Using PCA to initialize new task queries from a decorrelated subspace of old queries facilitates efficient learning of novel categories while preserving the query manifold structure.
  • Explicitly regularizing the correlation structure of instance queries encourages greater separability and consistency, which is vital for robust instance tracking in continual settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.