Reproduce “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, Kfir Aberman

Researchers at Google Research developed DreamBooth, a method for fine-tuning text-to-image diffusion models to generate novel images of a specific subject given only a few reference images. The approach uses rare-token identifiers and a class-specific prior preservation loss to maintain subject identity across diverse contexts and poses. DreamBooth quantitatively outperformed prior methods like Textual Inversion in subject fidelity and user studies, demonstrating capabilities such as recontextualization, novel view synthesis, and consistent character generation.

Key insights

  • Large text-to-image diffusion models can integrate new subject information effectively from just a few images without significant overfitting or catastrophic forgetting.
  • Using a coarse class noun alongside a unique rare-token identifier is crucial for leveraging the model's existing semantic knowledge, enabling the generation of novel poses and articulations.
  • The proposed class-specific prior preservation loss is effective in counteracting 'language drift' during fine-tuning, maintaining the model's ability to generate diverse instances of the broader class while embedding the specific subject.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.