Reproduce “DrVideo: Document Retrieval Based Long Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, Jianfei Cai
DrVideo addresses long video understanding by reframing it as a document retrieval and comprehension task for Large Language Models. Its iterative agent-based framework dynamically identifies and augments key visual information, achieving state-of-the-art performance across EgoSchema, MovieChat-1K, and Video-MME benchmarks.
Key insights
- Adaptive information augmentation using both general detailed descriptions and question-specific answers (VQA-type) is crucial for comprehensive video understanding.
- There exists an optimal number of initial key frames and iterative rounds; providing too much initial information or performing excessive iterations can introduce noise and degrade performance.
- The effectiveness of the document retrieval-based framework scales with the capabilities of the underlying foundation models, with proprietary LLMs like GPT-4 showing significantly higher performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.