Reproduce “MVEB: Massive Video Embedding Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Adnan El Assadi, Roman Solomatin, Isaac Chung, Chenghao Xiao, Deep Shah, Manan Dey, Shriya Sudhakar, Zacharie Bugaud and 8 more
MVEB (Massive Video Embedding Benchmark) introduces a comprehensive, unified benchmark of 23 diverse tasks for evaluating general-purpose video embedding models zero-shot, integrating with the broader MTEB ecosystem. The benchmark revealed that MLLM-based embedding models significantly outperform generative MLLMs used as embedders, and audio's contribution to video understanding is contingent on dataset annotation provenance, sometimes hindering performance.
Key insights
- Audio's contribution to video understanding varies significantly based on dataset provenance; it generally aids AV-grounded datasets but hurts V-grounded datasets, acting as a confounder if labels do not reward its presence.
- Performance scales logarithmically with the number of video frames sampled, with significant gains up to 8 frames, and returns plateauing beyond 32 frames, indicating a practical ceiling for general-purpose evaluation.
- Cross-modal retrieval capabilities are decoupled into distinct groups (text-target, video-target with audio in query, text-as-query), suggesting different strengths are needed for various retrieval directions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.