Reproduce “Video Action Differencing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
James Burgess, Xiaohan Wang, Yuhui Zhang, Anita Rau, Alejandro Lozano, Lisa Dunlap, Trevor Darrell, Serena Yeung-Levy
Stanford and UC Berkeley researchers introduce Video Action Differencing (VidDiff), a framework for automatically identifying and describing subtle differences between videos of the same action, along with VidDiffBench, a comprehensive dataset spanning fitness, sports, and medical domains that reveals current large multimodal models achieve only moderate performance on fine-grained action comparison tasks.
Key insights
- Accurate frame localization is crucial for performance, demonstrated through ablation studies
- Breaking down complex video comparison into subtasks using agentic workflows outperforms end-to-end approaches
- Current vision-language models struggle with subtle differences even given perfect frame alignment
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.