Reproduce “VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Darshana Saravanan, Varun Gupta, Darshan Singh, Zeeshan Khan, Vineet Gandhi, Makarand Tapaswi

VELOCITI introduces a new benchmark that uses seven diagnostic tests and a strict entailment metric to evaluate compositional reasoning in Video-Large Language Models (Video-LLMs) on short video clips. The evaluations reveal that state-of-the-art Video-LLMs achieve less than half of human accuracy, particularly struggling with action understanding, in-video negation, and temporal event chronology, which exposes fundamental gaps in their reasoning capabilities.

Key insights

  • State-of-the-art Video-LLMs, including powerful commercial models, demonstrate significant foundational weaknesses in compositional reasoning, achieving less than 50% accuracy on VELOCITI compared to 93% for humans.
  • Models exhibit specific difficulties with action understanding and discerning the correct temporal order of events, consistently performing worse on action-related and chronology tests.
  • Traditional evaluation metrics like ClassicVLE and multiple-choice setups can mask true reasoning gaps, as models often exhibit a 'Yes' bias or choice biases that inflate reported performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.