Reproduce “Beyond Simple Edits: Composed Video Retrieval with Dense Modifications” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Omkar Thawakar, Dmitry Demidov, Ritesh Thawkar, Rao Muhammad Anwer, Mubarak Shah, Fahad Shahbaz Khan, Salman Khan

Researchers from MBZUAI and collaborating institutions introduced Dense-WebVid-CoVR, a new large-scale dataset for Composed Video Retrieval (CoVR) featuring modification texts that are seven times more detailed than prior benchmarks, and proposed a unified fusion model architecture. Their approach achieved 71.26% Recall@1 on the new dataset, surpassing prior methods by 3.4%, and demonstrated a 3x faster inference speed.

Key insights

  • Dense, context-aware modification texts are crucial for training CoVR models to perform fine-grained retrieval, enabling a deeper understanding of visual and temporal changes.
  • A unified multimodal fusion approach, where query video, its description, and modification text are processed simultaneously, effectively captures rich contextual inter-dependencies and enhances retrieval performance and efficiency over pairwise fusion methods.
  • High-quality, large-scale dataset construction, leveraging the capabilities of advanced LLMs for detailed text generation and rigorous manual verification, is fundamental for advancing complex multimodal understanding tasks like CoVR.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.