Reproduce “Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhiyuan You, Zheyuan Li, Jinjin Gu, Zhenfei Yin, Tianfan Xue, Chao Dong

DepictQA introduces a language-based image quality assessment (IQA) framework that utilizes multi-modal language models to provide descriptive, human-like reasoning rather than just numerical scores. This approach achieves superior accuracy on comparison tasks, outperforming state-of-the-art score-based methods, and offers detailed explanations of image distortions and their impact.

Key insights

  • MLLMs can be effectively fine-tuned to move beyond quantitative IQA scores, enabling human-like descriptive and comparative reasoning about image quality.
  • A hierarchical task framework (description, comparison, reasoning) and a dedicated multi-modal dataset with detailed textual annotations are essential for training MLLMs to acquire comprehensive IQA abilities.
  • Specific strategies like using unique tags for multi-image inputs and combining multi-source training data are critical for MLLMs to accurately process and differentiate between multiple visual inputs for fine-grained low-level vision tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.