Reproduce “MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
De-An Huang, Zhiding Yu, Anima Anandkumar
MinVIS proposes a video instance segmentation framework that achieves competitive performance by training an image instance segmentation model on individual frames and performing temporal tracking through cosine similarity matching of query embeddings. This method significantly reduces annotation requirements and computational overhead, surpassing previous state-of-the-art results on YouTube-VIS 2019/2021 and achieving over 10% AP improvement on the challenging OVIS dataset with heavy occlusions.
Key insights
- Query embeddings, when properly trained for intra-frame instance discrimination, inherently possess sufficient temporal consistency for robust inter-frame tracking without dedicated video-based training.
- State-of-the-art video instance segmentation performance can be achieved through a purely image-based training approach combined with simple query matching, challenging the necessity of complex video-centric architectures.
- Online, frame-independent processing combined with query matching provides a computationally efficient and less memory-intensive alternative to processing entire video clips or using resource-heavy end-to-end video models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.