Reproduce “Video Colorization with Pre-trained Text-to-Image Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanyuan Liu, Minshan Xie, Jinbo Xing, Chengze Li, Tien-Tsin Wong
Researchers at The Chinese University of Hong Kong developed ColorDiffuser, a framework that adapts a pre-trained text-to-image diffusion model for video colorization, producing plausible, temporally consistent, and diverse results. The method achieves state-of-the-art quantitative performance, demonstrating improved color vividness and reduced flicker compared to existing techniques.
Key insights
- Pre-trained text-to-image diffusion models can be effectively adapted for video tasks by introducing specialized mechanisms to handle temporal coherence, circumventing the need for large-scale video-specific training.
- A novel Color Propagation Attention mechanism within the diffusion process, combined with an Alternated Sampling Strategy, enables robust and long-term temporal consistency in colorization by bi-directionally propagating color information.
- A specialized Video Colorization VQVAE decoder, which incorporates pixel-level grayscale features and pseudo-3D convolutions, is crucial for reconstructing high-quality, artifact-free, and temporally coherent colorized video from latent representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.