Reproduce “VISTA: Video Interaction Spatio-Temporal Analysis Benchmark” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alejandro Aparcedo, Akash Kumar, Aaryan Garg, Dalton Pham, Wen-Kai Chen, Anirudh Bharadwaj, Aman Chadha, Yogesh Rawat

VISTA introduces a large-scale, interaction-aware diagnostic benchmark to evaluate spatio-temporal understanding in Vision-Language Models (VLMs) across complex video interactions. Its evaluation of 11 state-of-the-art VLMs revealed systematic biases, including underperformance on same-entity interactions and tendencies toward "semantic-intent inflation" and "social-first bias."

Key insights

  • VLMs demonstrate systematic limitations in disambiguating visually similar instances of the same class, leading to consistent underperformance on same-entity interactions.
  • Models often struggle with open-ended freeform natural language queries compared to concise referral queries, indicating a reliance on explicit syntactic cues rather than robust multimodal reasoning from richer context.
  • Current VLMs exhibit biases such as "semantic-intent inflation" (over-interpreting high-intent or affective scenes) and a "social-first bias" (prioritizing social/affective cues over physical dynamics) in interaction understanding.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.