Reproduce “Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev and 14 more
Reka introduces Vibe-Eval, a new open evaluation benchmark for multimodal language models (MMLMs), featuring 269 visual understanding prompts, including a 'hard-set' derived from challenges faced by frontier models. This benchmark helps differentiate top-performing MMLMs, reveals phenomena like inverse scaling, and provides a framework for robustly assessing their evolving capabilities.
Key insights
- Observed 'inverse scaling' on some hard prompts, where smaller MMLMs surprisingly outperformed larger, frontier models, suggesting potential language bias or undertraining of the visual modality in larger models.
- The creation and accurate judgment of hard, open-ended multimodal prompts are inherently difficult, requiring prompts to be unambiguous, useful, and error-free, and evaluation to manage the subjectivity of partial credit.
- The 'hard' designation of prompts is temporary as models continuously improve, necessitating periodic extension of the benchmark to maintain its relevance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.