Reproduce “ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ali Athar, Xueqing Deng, Liang-Chieh Chen
The ViCaS dataset unifies holistic and pixel-level video understanding by providing 20,416 videos with detailed human-written captions and temporally consistent, pixel-accurate segmentation masks for referenced objects. It establishes a benchmark where the Video-LLaVA-Seg baseline model achieves 3.0 Caption Accuracy and 16.5 mAP for language-guided video instance segmentation on the test set, outperforming existing referral VOS approaches.
Key insights
- LLM-based evaluation metrics (e.g., Llama3-70B) demonstrate significantly higher correlation with human judgment for open-ended video caption scoring compared to embedding or classical word-matching methods.
- Sufficient temporal context, achieved through the 'slow-fast' pooling strategy in the vision backbone, is crucial for effective performance in both video captioning and language-guided video instance segmentation.
- Multi-task training for both captioning and LG-VIS tasks shows synergistic benefits, with pixel-level segmentation performance improving when concurrently learning holistic video understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.