Reproduce “Segment and Track Anything” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, Yi Yang
Segment and Track Anything (SAM-Track), developed by Zhejiang University's ReLER Lab, unifies video object segmentation and tracking by integrating SAM, Grounding-DINO, and their top-performing DeAOT model. The framework achieves competitive performance, such as 92.0% Avg J&F on DAVIS-2016 Val with user-friendly click-based initialization, and is capable of tracking newly appearing objects across various real-world scenarios.
Key insights
- Combining image-centric foundation models like SAM and Grounding-DINO with a specialized, high-performing video object tracker (DeAOT) effectively extends their capabilities for robust video understanding tasks.
- Multimodal user interaction methods (clicks, strokes, text) can significantly reduce the annotation burden for video object segmentation and tracking without compromising performance.
- An automatic mechanism for detecting and tracking newly appearing objects throughout a video sequence is critical for practical application in dynamic real-world scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.