Reproduce “UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling” on OpenResearch

About this paper

Haider Al-Tahan, Quentin Garrido, Randall Balestriero, Diane Bouchacourt, Caner Hazirbas, Mark Ibrahim

UniBench introduces a unified evaluation framework integrating over 50 Vision-Language Model (VLM) benchmarks, providing a standardized tool to assess capabilities and identify limitations. The study evaluates nearly 60 VLMs, revealing that scaling model size or training data offers minimal improvements for visual reasoning and relational understanding, while also exposing surprising underperformance on fundamental tasks like digit recognition.

Key insights

• Scaling model size and training data offers minimal benefits for VLM visual reasoning and relational understanding tasks, unlike other capabilities.
• Despite advanced architectures, state-of-the-art VLMs exhibit surprising underperformance on foundational tasks like digit recognition (e.g., MNIST) and counting.
• Beyond a certain point, data quality and targeted learning objectives significantly contribute to VLM performance, often outperforming gains from increased scale alone.

Summary by alphaXiv.