Reproduce “MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhichao Wei, Qingkun Su, Long Qin, Weizhi Wang
MM-Diff is a tuning-free framework that achieves high-fidelity personalized image generation for both single and multiple subjects by integrating multi-modal conditions and employing cross-attention map constraints. It generates images in seconds, significantly outperforming prior tuning-free methods in subject fidelity and effectively resolving attribute binding issues in multi-subject scenarios.
Key insights
- Augmenting text embeddings with global visual (CLS) embeddings, processed by LoRA, effectively enhances both text consistency and subject fidelity.
- Distilling detailed visual information into a compact set of subject embeddings via a transformer-decoder (SE-Refiner) efficiently injects rich visual cues without high computational overhead.
- Explicitly constraining cross-attention maps to ground-truth segmentation masks during training disentangles attribute assignments for multiple subjects, resolving the attribute binding issue without needing manual layouts during inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.