Reproduce “Personalized Residuals for Concept-Driven Text-to-Image Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cusuh Ham, Matthew Fisher, James Hays, Nicholas Kolkin, Yuchen Liu, Richard Zhang, Tobias Hinz
Researchers from Georgia Institute of Technology and Adobe Research developed Personalized Residuals, an efficient method for text-to-image personalization that adapts specific layers of diffusion models with only 1.2 million parameters. This approach eliminates the need for regularization images and, combined with Localized Attention-Guided (LAG) Sampling, produces high-quality, concept-driven images while preserving identity and offering flexible recontextualization.
Key insights
- Applying LoRA-based residuals to output projection layers (1x1 convolutions) is highly efficient for concept personalization and naturally avoids the need for regularization images.
- Cross-attention maps within diffusion models can be effectively leveraged during inference to dynamically mask and blend features, enabling localized application of personalized concepts and mitigating overfitting.
- Human preference studies indicate that the method excels at identity preservation and recontextualization, often surpassing baselines despite sometimes having slightly lower automated text alignment scores.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.