Reproduce “DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhendong Wang, Jianmin Bao, Shuyang Gu, Dong Chen, Wengang Zhou, Houqiang Li

This work introduces DesignDiffusion, a diffusion model framework designed for high-quality text-to-design image generation that accurately renders specific text within images. It achieves an FID score of 19.87 and an F-score of 0.862 for text rendering, producing visually coherent designs with legible text in a single generation step.

Key insights

  • Prompt enhancement provides explicit character-level information to the CLIP text encoder, guiding the model to focus on individual letters for rendering.
  • A character localization loss encourages character embeddings to spatially localize to specific regions in the image, ensuring correct order and spacing.
  • Self-Play Direct Preference Optimization (SP-DPO) refines the model by learning from self-generated 'losing' samples and 'winning' reference images, improving text rendering quality without human feedback.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.