Reproduce “Simple Baselines for Interactive Video Retrieval with Questions and Answers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kaiqu Liang, Samuel Albanie

Researchers from Princeton University and the University of Cambridge developed effective baselines for interactive video retrieval that leverage video question-answering models to simulate user interactions. This approach significantly enhances retrieval performance, achieving up to 67% R@1 after 6 interaction rounds on the MSR-VTT dataset, while reducing the need for costly real-world dialogue data collection.

Key insights

  • Simulating user interactions through VideoQA models is an effective approach for developing and evaluating interactive video retrieval systems, circumventing the need for extensive real-world dialogue data.
  • Heuristic-based question generation strategies, particularly those focusing on objects, actions, and locations, prove more effective than large language model-based (Auto-text/Auto-text-vid) approaches for query refinement.
  • Interactive retrieval consistently improves performance as the number of interaction rounds increases, demonstrating the value of iterative user feedback in narrowing down search results.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.