Reproduce “Text-to-Image Diffusion Models are Zero-Shot Classifiers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kevin Clark, Priyank Jaini

Researchers from Google DeepMind demonstrated that text-to-image diffusion models, such as Imagen and Stable Diffusion, function effectively as zero-shot image classifiers by leveraging their denoising objective as a proxy for conditional likelihood. The study found these generative models achieve competitive classification accuracy against contrastively trained models like CLIP, exhibit state-of-the-art robustness to shape-texture conflicts, and show improved attribute binding capabilities.

Key insights

  • Generative pre-training, as seen in text-to-image diffusion models, yields powerful discriminative capabilities that are competitive with, and in some areas superior to, contrastive pre-training.
  • Diffusion models exhibit a strong shape bias, outperforming all prior methods on shape-texture conflict datasets, likely due to their denoising process focusing on underlying object structures.
  • The integration of text and image information via cross-attention mechanisms and powerful text encoders (e.g., T5 in Imagen) in diffusion models contributes to their superior attribute binding compared to models with independent encoders like CLIP.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.