Reproduce “Visual Reasoning Evaluation of Grok, Deepseek Janus, Gemini, Qwen, Mistral, and ChatGPT” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nidhal Jegham, Marwan Abdelatti, Abdeltawab Hendawi
A comprehensive evaluation assessed state-of-the-art multimodal Large Language Models, focusing on multi-image reasoning, response consistency, uncertainty calibration, and bias susceptibility. The study found that ChatGPT-o1 achieved the highest overall accuracy, while QVQ-72B-Preview showed the strongest ability to identify unanswerable questions, and some models exhibited significant positional biases as revealed by entropy-based consistency analysis.
Key insights
- Model size does not directly correlate with performance, as some larger models underperformed compared to smaller ones, suggesting architectural design and training methodology are more critical.
- Models like the Janus series exhibited high entropy, indicating unstable reasoning and susceptibility to positional biases in answer choices, a critical weakness traditional accuracy metrics might miss.
- There are distinct performance profiles across visual reasoning tasks, with some models excelling in diagram understanding or image-text matching, while others struggled with geographic understanding or multi-image integration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.