Reproduce “MagicEraser: Erasing Any Objects via Semantics-Aware Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fan Li, Zixiao Zhang, Yi Huang, Jianzhuang Liu, Renjing Pei, Bin Shao, Songcen Xu
MagicEraser introduces a diffusion model-based framework for object erasure, employing semantics-aware control and a novel training strategy to generate high-fidelity, coherent backgrounds without requiring manual text prompts. It consistently outperforms established inpainting algorithms and commercial tools in quantitative and qualitative evaluations.
Key insights
- Pre-initializing masked regions with a traditional inpainting model before diffusion helps to safely use higher denoising strengths, preventing undesired artifacts while maintaining texture harmony.
- A learned placeholder token for 'background completion' combined with automatic background-semantic-based prompt construction offers a user-friendly and effective way to guide diffusion models for object erasure.
- Directly modulating the diffusion model's self-attention layers with semantic cues from panoptic segmentation can effectively guide content generation towards background restoration and away from foreground object creation.
- Training a model to restore an original image from one where an object has been shifted and blended into its background region (OLRD) is a highly effective strategy for learning object erasure.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.