Reproduce “ControlCom: Controllable Image Composition using Diffusion Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bo Zhang, Yuxuan Duan, Jun Lan, Yan Hong, Huijia Zhu, Weiqiang Wang, Li Niu
ControlCom introduces a diffusion-based model for controllable image composition, allowing fine-grained adjustment of foreground illumination and pose while maintaining subject identity. The method achieves superior foreground fidelity, with a CLIP_fg of 88.31 on COCOEE for generative composition, and unifies multiple composition tasks within a single framework.
Key insights
- A simple 2-dimensional indicator vector can effectively provide fine-grained, selective control over complex generative tasks, such as preserving or adjusting foreground illumination and pose.
- A multi-stage conditioning approach, which first applies global semantic guidance and then refines with local, feature-modulated attention, is crucial for achieving high foreground fidelity and preserving identity in generative composition.
- A self-supervised training framework with strategically designed attribute-specific augmentations can overcome the lack of large-scale, multi-task annotated datasets for complex, controllable generative image composition.
- Integrating the conditioning information, especially the indicator map, directly into early U-Net layers facilitates explicit control over attribute modifications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.