Reproduce “HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, Chaim Baskin
This research introduces HERBench, a new benchmark for Video Question Answering (VideoQA) specifically designed to assess multi-evidence integration in Video Large Language Models (Video-LLMs). It reveals that state-of-the-art Video-LLMs exhibit systematically low performance, achieving only 38.2% mean accuracy, due to distinct evidence retrieval and fusion deficits, even when provided with necessary information.
Key insights
- Performance limitations of Video-LLMs on multi-evidence integration tasks can be disentangled into two bottlenecks: an evidence retrieval deficit (failure to identify all necessary frames) and an evidence aggregation (fusion) deficit (failure to effectively combine retrieved evidence).
- Even with perfect access to relevant ground-truth frames, Video-LLMs struggle to integrate multiple cues, often exhibiting an over-concentration of importance on a single frame in incorrect predictions.
- HERBench, with a mean MRFS of 5.49, imposes a uniquely high evidential demand compared to prior benchmarks, and model performance inversely correlates with this evidential burden (R² = 0.82).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.