Reproduce “VGI-Bench: Probing Visual Intelligence in Video Generation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuan He, Cong Wei, Yuhao Cheng, Linrui Ma, Yuxuan Zhang, Zuojun Li, Yuhao Wen, Jize Jiang and 15 more
VGI-Bench introduces a new benchmark to evaluate the visual intelligence and reasoning capabilities of video generation models, utilizing photorealistic inputs and process-sensitive tasks. Evaluations on contemporary models show that even top-performing systems like Seedance 2.0 achieve moderate overall scores (51.0%), revealing significant challenges in physical causality, rule following, and temporal consistency.
Key insights
- State-of-the-art video generation models exhibit significant limitations in visual reasoning, particularly in tasks requiring multi-step rule following, maintaining object topology, and coherent temporal state tracking.
- The VLM-based evaluation approach demonstrated reliability, achieving an AUC of 0.803 and a pairwise accuracy of 73.2% when compared against human annotations, confirming its effectiveness in assessing video outputs.
- Analysis of denoising trajectories in diffusion models indicates limited 'self-correction,' with reasoning errors rarely being fundamentally rectified after early stages, suggesting initial hypotheses are mostly refined rather than significantly altered.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.