Reproduce “ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Philip Schroeder, Ondrej Biza, Thomas Weng, Hongyin Luo, James Glass
ROVER, a recursive framework for vision-language models, improves reasoning over long video sequences in embodied tasks by decomposing them into subtasks. It achieves higher accuracy in task progress estimation and natural language reasoning, while significantly reducing computational cost and mitigating hallucinations by using a localized sliding context window.
Key insights
- Recursive decomposition of tasks allows VLMs to process complex, long-horizon videos by focusing on manageable subtasks, which improves reasoning accuracy and context management.
- A compact, subtask-specific sliding window for visual context is effective in reducing VLM hallucinations and improving computational efficiency, especially during unexpected task deviations.
- Evaluating VLM reasoning on a diverse range of non-expert trajectories, rather than just optimal paths, is crucial for assessing real-world robustness and identifying model limitations in embodied tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.