Reproduce “RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yonglin Li, Jing Zhang, Xiao Teng, Long Lan, Xinwang Liu
RefSAM, developed by researchers at NUDT and the University of Sydney, introduces the first end-to-end framework to adapt the Segment Anything Model (SAM) for Referring Video Object Segmentation. It achieves state-of-the-art performance on Ref-DAVIS17 with a 71.9 J&F score (a 7.1-point increase) and Refer-Youtube-VOS with a 67.6 J&F score (a 1.6-point increase) by integrating lightweight language understanding and temporal tracking modules.
Key insights
- Foundation models like SAM can be effectively adapted for complex multimodal video tasks by integrating lightweight, fine-tuned modules.
- Dedicated cross-modal fusion (Cross-Modal MLP, HDA) is essential to bridge the semantic gap between language and vision for precise segmentation.
- Implicit tracking through learned tokens from previous frames significantly enhances spatiotemporal consistency in video object segmentation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.