Reproduce “Visual Enumeration Remains Challenging for Multimodal Generative AI” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alberto Testolin, Kuinan Hou, Marco Zorzi
An investigation into multimodal generative AI models reveals persistent struggles with visual enumeration tasks, demonstrating that state-of-the-art systems primarily rely on approximation rather than systematic counting. The work identifies biases in training data, specifically power-law distributions of numerosity and noisy image-text alignments, as key factors hindering the development of robust numerical abilities in these models.
Key insights
- Multimodal generative AI models consistently exhibit approximation behaviors, similar to human subitizing for small numbers and Weber's law for larger ones, rather than performing exact, systematic counting.
- The power-law distribution of numerical mentions in training data, which heavily over-represents small numbers and under-represents larger ones, along with noisy image-text alignments, significantly impede the acquisition of robust enumeration skills.
- Simply scaling up model size does not guarantee improved or systematic visual enumeration capabilities; some larger or newer proprietary models showed comparable or even degraded performance compared to smaller or older counterparts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.