Reproduce “Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun and 3 more
Q-BENCH introduces the first systematic benchmark to evaluate Multi-modality Large Language Models (MLLMs) on low-level visual perception, description, and quality assessment. The benchmark reveals that while MLLMs possess preliminary low-level visual skills, they face challenges in perceiving technical distortions and generating precise descriptions, yet can outperform traditional metrics in quantitative quality assessment.
Key insights
- MLLMs demonstrate an inherent, preliminary ability to perceive and understand low-level visual attributes, even without explicit fine-tuning for these tasks.
- The main bottleneck for MLLMs in low-level description is often accurate perception of attributes rather than the ability to articulate them in natural language.
- Robust, model-agnostic evaluation strategies are crucial for benchmarking MLLMs on nuanced tasks that require both precise textual and quantitative outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.