Reproduce “Improved Diffusion-based Image Colorization via Piggybacked Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanyuan Liu, Jinbo Xing, Minshan Xie, Chengze Li, Tien-Tsin Wong
This paper presents a method for image colorization by adapting large-scale, pre-trained Text-to-Image diffusion models, such as Stable Diffusion, instead of training from scratch. The approach utilizes a Latent Diffusion Guider and a Lightness-aware VQVAE to produce realistic and semantically consistent colorizations while achieving state-of-the-art quantitative results and enabling controllable colorization via hints and text prompts.
Key insights
- Large-scale pre-trained Text-to-Image (T2I) diffusion models contain rich semantic understanding and color priors that can be adapted for image colorization without extensive retraining.
- Decoupling the colorization task into a latent diffusion guiding process and a lightness-aware reconstruction ensures both high-quality color generation and pixel-perfect structural preservation.
- Adapting foundation models via lightweight modules offers significant computational efficiency and effective knowledge transfer compared to training models from scratch.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.