Reproduce “VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Divake Kumar, Sina Tayebati, Devashri Naik, Ranganath Krishnan, Amit Ranjan Trivedi
The method adds a post-hoc conformal predictor to a vision-language model judge, using rating-token probabilities and human-rated calibration examples to produce score intervals without retraining the judge. In ChartQA, LLaVA-Critic-7B has the highest Pearson correlation across 14 tasks, .507, but a mean interval width of 3.08 points on a 1–5 scale, showing that correlation does not ensure precise absolute scores.
Key insights
- VLM judge uncertainty is highly task-dependent, with prediction interval widths varying up to 68% across tasks, providing a quantitative reliability map for multimodal evaluation domains.
- A "ranking-scoring decoupling" phenomenon was identified, where VLM judges can achieve high correlation in ranking responses but simultaneously produce unreliable absolute scores (wide prediction intervals).
- The quality of ground truth annotations significantly influences uncertainty estimates; well-defined tasks with multi-annotator labels yield substantially tighter and more actionable intervals (e.g., 4.5x narrower).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.