Reproduce “Generating images of rare concepts using pre-trained diffusion models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, Gal Chechik
The paper introduces SeedSelect, a method to improve the generation of images for rare or structurally complex concepts using pre-trained text-to-image diffusion models. It optimizes the initial noise seed to guide the model, achieving significantly higher classification accuracy for rare concepts (e.g., 76.1% for ImageNet tail classes) while maintaining image realism and diversity.
Key insights
- Pre-trained diffusion models implicitly contain knowledge of rare concepts, but this knowledge is only accessible via specific, narrow regions of the initial noise latent space.
- Optimizing the initial noise seed, guided by a small set of real reference images, can effectively unlock this latent knowledge for under-represented concepts.
- A balanced loss combining semantic consistency (using CLIP embeddings) and natural appearance (using VAE-encoded latents) is crucial for generating both faithful and realistic images.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.