Reproduce “ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dingkun Yan, Liang Yuan, Erwin Wu, Yuma Nishioka, Issei Fujishiro, Suguru Saito
ColorizeDiffusion presents a method for adjustable anime-style sketch colorization that integrates reference images and text prompts. It addresses a "distribution problem" in dual-conditioned generation, yielding superior semantic fidelity and segmentation compared to existing methods, while enabling zero-shot text-based manipulation of visual attributes.
Key insights
- The "distribution problem" represents a fundamental conflict in multi-modal generative tasks when different conditional inputs contain potentially conflicting semantic and spatial information.
- Strategic noise injection or condition dropping during training can effectively regularize diffusion models, preventing overfitting to reference features and preserving input sketch fidelity.
- Image-guided diffusion models can achieve robust, fine-grained, and sequential text-based manipulation by leveraging CLIP's latent space, extending capabilities typically associated with text-to-image models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.