Reproduce “Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eric Slyman, Mehrab Tanjim, Kushal Kafle, Stefan Lee
Researchers at Adobe and Oregon State University developed Multimodal Mixture-of-Bayesian Prompt Ensembles (MMB) to improve the reliability of MLLM-as-a-judge systems for text-to-image evaluation. This method integrates visual context via image clustering to dynamically weight prompts, achieving a 36% reduction in Expected Calibration Error and a 1.8% F1 score improvement over baseline ensemble methods on the HPSv2 benchmark.
Key insights
- The effectiveness of a judge prompt in multimodal settings is contingent on the visual characteristics of the input, necessitating a multimodal-aware approach for prompt weighting.
- Dynamically assigning prompt weights based on image clusters significantly enhances both the calibration and discriminative accuracy of MLLM judges for TTI evaluation.
- Improved calibration in MLLM judges enables practical applications such as selective classification, where high-confidence judgments can be automated and low-confidence ones deferred to human reviewers, optimizing evaluation costs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.