Reproduce “VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinho Park, Youbin Kim, Hogun Park, Eunbyung Park
VGenST-Bench introduces a novel video benchmark for spatio-temporal reasoning in Multimodal Large Language Models (MLLMs), constructed by actively synthesizing photorealistic videos with generative models. Evaluations reveal top MLLMs are over 13 percentage points below human performance on complex reasoning tasks, showing significant degradation from basic perception to higher-order inference.
Key insights
- The active synthesis approach effectively mitigates data contamination and shortcut exploitation, providing a more reliable and diagnostic testbed for MLLM spatio-temporal reasoning.
- MLLMs exhibit a consistent and substantial performance degradation as cognitive tasks move from basic visual perception (L1) to higher-order spatio-temporal reasoning (L3), highlighting a systematic bottleneck in complex inference.
- MLLMs struggle significantly with 'None-of-these' answer options and open-ended questions, indicating a reliance on choice priors and positional biases rather than robust reasoning in standard multiple-choice formats.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.