Reproduce “IA-T2I: Internet-Augmented Text-to-Image Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chuanhao Li, Jianwen Sun, Yukang Feng, Mingliang Zhai, Yifan Chang, Kaipeng Zhang

The IA-T2I framework enhances text-to-image generation by dynamically retrieving real-time internet images to address uncertain knowledge in prompts, improving accuracy for rare, unknown, or ambiguous content. It achieves an 81.2% human preference over raw T2I models and utilizes a GPT-4o-based automatic evaluation method.

Key insights

  • Dynamically augmenting text-to-image (T2I) models with real-time internet-retrieved images improves their ability to handle uncertain, rare, or unknown knowledge present in prompts.
  • A hierarchical image selection process, combining diversity selection via clustering and relevance re-ranking with large vision-language models (LVLMs), is effective for filtering noisy web search results into useful reference images.
  • An iterative self-reflection mechanism, which evaluates generated images based on instruction following and reference helpfulness, allows for refinement and improved accuracy over multiple generation rounds.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.