Reproduce “Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation” on OpenResearch

About this paper

Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, Minjoon Seo

Prometheus-Vision evaluates vision-language responses by seeing the image, instruction, response, rubric, and reference answer, then producing feedback and a score from 1 to 5. It is trained on the Perception Collection of 150,000 scored responses. On 60 LLaVA-Bench instances, its 13B version reaches 0.786 Pearson correlation with GPT-4V, versus −0.005 for LLaVA-1.5 13B, indicating tracking in that setting, not universal judging accuracy.

Key insights

• An open-source VLM can effectively function as a fine-grained judge, demonstrating evaluative capabilities comparable to human annotators and state-of-the-art proprietary VLMs.
• A meticulously constructed multimodal feedback dataset with specific, fine-grained rubrics is crucial for training a VLM to perform nuanced evaluations, moving beyond generic quality assessments.
• The 'VLM-as-a-Judge' paradigm provides a scalable, cost-effective, and transparent alternative to human evaluation or proprietary models, democratizing access to high-quality VLM assessment.

Summary by alphaXiv.