Reproduce “Multi-Concept Customization of Text-to-Image Diffusion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu

Custom Diffusion introduces an efficient fine-tuning method for text-to-image diffusion models, enabling the acquisition and compositional generation of multiple user-defined concepts from limited examples. It achieves this by selectively updating key and value projection matrices in cross-attention layers, requiring only 75MB storage per concept and demonstrating strong qualitative and quantitative performance in personalizing image generation.

Key insights

  • Cross-attention layer's key and value projection matrices (Wk, Wv) are the most critical parameters for concept learning, exhibiting the highest parameter changes during full model fine-tuning.
  • Selective fine-tuning of these specific matrices, combined with a modifier token (V*) and regularization with real images, can efficiently embed new concepts without compromising prior knowledge or increasing model size significantly.
  • Effective compositional generation of multiple new concepts can be achieved by either jointly training all concepts or by merging separately learned concept models via a closed-form constrained optimization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.