Reproduce “Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ahmad Jarjis Hasan, Jonathan Lwowski
The authors audit how benchmark scoring rules affect measured accuracy when a model’s response cannot be parsed. On identical stored outputs, they compare fallback rules, including snapping to the closest option and abstaining, and examine whether responses commit to an answer. On MMAD, Pixtral-12B scores 0.656 versus 0.444, a 21.2-point gap; this reflects scoring, not changed outputs or capability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.