Reproduce “GenCeption: Evaluate Vision LLMs with Unlabeled Unimodal Data” on OpenResearch

About this paper

Lele Cao, Valentin Buchner, Zineb Senane, Fangkai Yang

GenCeption is an annotation-free evaluation framework for Vision Large Language Models (VLLMs) that quantifies inter-modality semantic coherence and hallucination propensity by iteratively describing and generating images from unlabeled unimodal data. The method achieved strong correlations with established benchmarks like OpenCompass and HallusionBench, while also revealing a notable performance gap between current VLLMs and human annotators, especially in text-intensive visual reasoning tasks.

Key insights

• Evaluating VLLMs via iterative description-generation cycles provides a cost-effective, annotation-free, and robust method for measuring inter-modal semantic coherence and hallucination.
• The continuous nature of the GC@T metric offers a more nuanced and enduring evaluation standard, helping to mitigate benchmark saturation and differentiate performance among highly capable models.
• The inherent design of GenCeption's iterative process compels VLLMs to genuinely engage with and understand the visual input, ensuring the relevance of the non-textual modality in the evaluation.

Summary by alphaXiv.