Reproduce “ImageDoctor: Diagnosing Text-to-Image Generation via Grounded Image Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxiang Guo, Jiang Liu, Ze Wang, Hao Chen, Ximeng Sun, Yang Zhao, Jialian Wu, Xiaodong Yu and 2 more
The ImageDoctor framework introduces a unified, multi-aspect evaluation for text-to-image (T2I) generation, providing both scalar quality scores and pixel-level flaw heatmaps via grounded image reasoning. It leverages this dense, spatially aware feedback to enable a novel DenseFlow-GRPO method, which significantly enhances T2I model performance and human preference alignment in reinforcement learning from human feedback (RLHF).
Key insights
- Combining multi-aspect scalar scores with pixel-level flaw heatmaps provides a comprehensive and interpretable diagnostic tool for T2I models, surpassing the limitations of single-score metrics.
- Dense, spatially grounded rewards derived from heatmaps are more effective for fine-tuning T2I models via RLHF than traditional sparse, image-level rewards, enabling targeted local refinements.
- A "look-think-predict" reasoning paradigm, inspired by human diagnostic processes and supported by MLLMs, can significantly improve the accuracy and interpretability of automated image evaluation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.