Reproduce “Continual Personalization for Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yu-Chien Liao, Jr-Jen Chen, Chi-Pin Huang, Ci-Siang Lin, Meng-Lin Wu, Yu-Chiang Frank Wang

Researchers from National Taiwan University and Qualcomm Technologies developed Concept Neuron Selection (CNS), a fusion-free framework for continually personalizing diffusion models. It identifies and incrementally finetunes concept-specific neurons, mitigating catastrophic forgetting and achieving superior multi-concept generation with zero additional memory or processing time during inference, updating only about 0.13% of total model parameters per concept.

Key insights

  • Specific neurons within diffusion model cross-attention layers can be identified and selectively finetuned to encode target concepts, indicating a localized responsibility for learned knowledge.
  • Distinguishing concept-specific neurons from general neurons is crucial for effective personalization, allowing targeted updates without compromising the model's broader generation capabilities.
  • A fusion-free approach to multi-concept personalization is achievable by directly modifying the base model's sparse concept neurons, significantly reducing inference-time computational overhead and memory storage.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.