Reproduce “FeVOS: Foresight Expression Video Object Segmentation” on OpenResearch

About this paper

Kehan Lan, Kaining Ying, Henghui Ding

Researchers from Fudan University introduced Foresight Expression Video Object Segmentation (FeVOS), a new task requiring pixel-level segmentation of objects based on anticipated future events. Their FeVOS-R1 model, employing a two-stage training strategy with Chain-of-Thought supervised fine-tuning and reinforcement learning, achieved 42.3 J&F on the new FeVOS dataset, improving upon a fine-tuned baseline by 6.5 points, and demonstrated improved generalization on existing RVOS benchmarks.

Key insights

• The two-stage training pipeline (CoT-SFT and RL) is complementary, with SFT establishing foundational reasoning and output format, and RL optimizing for direct segmentation performance, yielding substantial gains (+5.1 J&F over SFT-only).
• Explicit Chain-of-Thought annotations are crucial for guiding MLLMs to develop interpretable reasoning capabilities for predictive visual grounding tasks.
• Models trained on predictive reasoning tasks like FeVOS can demonstrate enhanced generalization and reasoning capabilities on traditional observational RVOS benchmarks, even surpassing zero-shot and fine-tuned baselines (e.g., +1.2 J&F on ReVOS vs. zero-shot).

Summary by alphaXiv.