Reproduce “Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hyungjin Kim, Seokho Ahn, Young-Duk Seo

Researchers from Inha University developed DrUM, a personalized Text-to-Image (T2I) generation framework that uses condition-level modeling in the latent space. It integrates user preferences via a lightweight, transformer-based adapter, achieving superior personalization quality and fine-grained control across various T2I models without requiring additional fine-tuning.

Key insights

  • Shifting personalized generation from prompt-level to condition-level modeling in the latent space effectively overcomes input token capacity limitations and enables more nuanced preference integration.
  • Coreset sampling provides an efficient method to accurately profile user preferences from extensive historical data, reducing computational burden while maintaining representativeness.
  • A lightweight, transformer-based adapter (PeCA) can seamlessly integrate personalized conditions into existing T2I models and text encoders without requiring expensive fine-tuning of the foundation models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.