Reproduce “LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models” on OpenResearch

About this paper

Lu Liu, Huiyu Duan, Chenxin Zhu, Jintong Lu, Haoyun Jiang, Liu Yang, Qiang Hu, Guangtao Zhai and 1 more

The research introduces LL-Bench, a comprehensive human-preference benchmark for evaluating large-scale generative models across 16 low-level vision tasks, and develops LL-Score, an MLLM-based evaluator. LL-Bench reveals that large generative models generally match or exceed conventional methods but exhibit significantly higher hallucination rates, while LL-Score achieves a Spearman Rank-order Correlation Coefficient of 0.6600 for quality assessment and high accuracy for hallucination detection, aligning closely with human judgments.

Key insights

• Large-Scale Generative Models (LGMs) show strong generalization and competitive performance in weather/occlusion removal, but their capabilities are uneven across other low-level vision tasks, exhibiting significant model- and task-specific variability.
• LGMs consistently produce substantially higher hallucination rates (e.g., lowest LGM rate 7.9% versus ~4% for conventional methods), with common types including color shifts, object hallucinations, and structural edits, indicating a need for enhanced fidelity control.
• Existing full-reference and no-reference IQA metrics show poor correlation with human preferences for generative low-level vision, underscoring the necessity for new evaluation approaches that can holistically assess perceptual quality and content fidelity.

Summary by alphaXiv.