Reproduce “CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tuna Han Salih Meral, Enis Simsar, Federico Tombari, Pinar Yanardag

CONFORM, a training-free framework from Virginia Tech and ETH Zürich, enhances the fidelity of text-to-image diffusion models by employing a contrastive learning objective on attention maps. This method consistently improves generated images by mitigating missing objects, correcting attribute binding errors, and improving object counting accuracy across various benchmarks and pre-trained models like Stable Diffusion and Imagen.

Key insights

  • Semantic fidelity issues in text-to-image models can be effectively addressed by intuitively promoting the segregation of distinct objects within attention maps while reinforcing object-attribute associations.
  • Contrastive learning, specifically InfoNCE loss, can serve as a unified objective to guide attention map formation, inherently addressing multiple fidelity problems without requiring separate, tailored functions for each failure mode.
  • Maintaining consistency of attention maps for the same token across backward diffusion timesteps is crucial to prevent attention scattering and preserve conceptual coherence throughout the generation process.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.