Reproduce “Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siyu Zou, Jiji Tang, Yiyi Zhou, Jing He, Chaoyi Zhao, Rongsheng Zhang, Zhipeng Hu, Xiaoshuai Sun
InstDiffEdit introduces an efficient, training-free method for diffusion-based image editing that instantly generates semantic masks from cross-modal attention maps. It achieves 5-6x faster inference and improves mask accuracy by 70.3% while enabling more precise localized edits compared to prior automatic mask generation techniques.
Key insights
- The inherent cross-modal attention mechanisms in pre-trained diffusion models contain rich semantic information that can be effectively extracted for automatic and instant mask generation.
- An adaptive, training-free aggregation of attention maps, using similarity to a reference token and thresholding, can refine noisy attention into clean, precise semantic masks.
- Combining mask-guided denoising with a final inpainting step using the refined mask enhances localized editing, mitigates artifacts, and maintains global image coherence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.