Reproduce “Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models” on OpenResearch

About this paper

Lin Zhu, Xinbing Wang, Chenghu Zhou, Qinying Gu, Nanyang Ye

Researchers from Shanghai Jiao Tong University developed a training-free, masking-based method to effectively decouple content from style in text-to-image diffusion models, mitigating content leakage while preserving style and text fidelity. The approach, built on the IP-Adapter framework, achieved the highest human preference score (36.7% on a custom dataset, 42.1% on StyleBench) and superior content leakage reduction compared to state-of-the-art methods.

Key insights

• The principle of 'less is more' holds, demonstrating that guiding diffusion models with appropriately selected fewer conditions (by masking irrelevant feature elements) can more effectively avoid unwanted content leakage and enhance style transfer.
• Masking elements within the image feature space (direct manipulation) is theoretically and empirically superior to cross-modal feature subtraction for content-style disentanglement, as it avoids image-text misalignment.
• Learning content directly from image features (Image-Adapter) is more effective for disentanglement than learning from content text features (Text-Adapter), even in tuning-based scenarios.

Summary by alphaXiv.