Reproduce “SpatialViz-Bench: A Cognitively-Grounded Benchmark for Diagnosing Spatial Visualization in MLLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siting Wang, Minnan Pei, Luoyang Sun, Cheng Deng, Yuchen Li, Kun Shao, Zheng Tian, Haifeng Zhang and 1 more
SpatialViz-Bench, a new cognitively-grounded benchmark, evaluates the spatial visualization capabilities of MLLMs through 12 programmatically generated tasks across four sub-abilities. It reveals substantial performance gaps between models and humans, with top proprietary models achieving up to 44.66% accuracy, and identifies fundamental perceptual and spatial transformation errors as dominant failure modes in MLLMs.
Key insights
- MLLMs exhibit significant limitations in fundamental visuospatial acquisition and transformation, with perceptual and spatial transformation errors accounting for nearly 60% of failures.
- A "CoT paradox" was observed where Chain-of-Thought prompting benefited top closed-source MLLMs but often degraded the accuracy of leading open-source models on pure-visual spatial tasks.
- Simply scaling MLLM parameters, as seen between 7B and 72B models, does not substantially resolve core spatial reasoning deficits, indicating a need for targeted architectural or training innovations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.