Reproduce “R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kaijie Chen, Zihao Lin, Zhiyang Xu, Ying Shen, Yuguang Yao, Joy Rimchala, Jiaxin Zhang, Lifu Huang

Researchers from UC Davis, Tongji University, Virginia Tech, and Intuit AI Research developed R2I-Bench, a comprehensive benchmark to evaluate reasoning capabilities in text-to-image models. Their evaluation showed open-source models scored below 45% on reasoning tasks and struggled particularly with mathematical reasoning, while closed-source models performed substantially better.

Key insights

  • Current open-source T2I models demonstrate limited reasoning capabilities, often treating prompts as keyword collections rather than applying logical understanding.
  • Mathematical reasoning presents a persistent and severe bottleneck for all evaluated T2I models.
  • Reasoning-enhanced techniques offer only marginal improvements, suggesting fundamental architectural or training data limitations in open-source models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.