Reproduce “EgoVLM: Policy Optimization for Egocentric Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ashwin Vinod, Shrey Pandit, Aditya Vavre, Linshen Liu
Researchers from The University of Texas at Austin develop EgoVLM, a reasoning-focused vision-language model that applies Group Relative Policy Optimization (GRPO) directly to egocentric video understanding without requiring chain-of-thought supervised fine-tuning, achieving state-of-the-art performance on EgoSchema benchmark while outperforming larger models like Qwen2.5-VL-7B (+13.87 accuracy points) and proprietary systems including GPT-4v and Gemini 1.5-Pro through explicit reasoning trace generation using a rule-based reward function combining format adherence and accuracy scoring, with the 3B parameter model demonstrating superior long-horizon planning capabilities on EgoPlan benchmark and maintaining competitive generalization to third-person video tasks on MV-Bench despite being optimized specifically for first-person perspective understanding.
Key insights
- GRPO can successfully instill sophisticated reasoning capabilities in VLMs without chain-of-thought supervised fine-tuning, reducing data dependency and training complexity
- Domain-specific optimization with reinforcement learning significantly outperforms general-purpose models on egocentric tasks while maintaining generalization to other video domains
- Explicit reasoning traces are crucial for egocentric benchmark performance but less beneficial for perceptual tasks, highlighting task-specific reasoning requirements
- Current VLM architectures struggle with temporal grounding and frame distinction, requiring novel input formats for improved frame-level representation
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.