Reproduce “Zero-Shot Video Restoration and Enhancement with Text-to-Image Latent Diffusion Models and Multi-Modal References” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cong Cao, Huanjing Yue, Xin Liu, Jingyu Yang
ZVRM, a framework for zero-shot video restoration and enhancement, adapts text-to-image latent diffusion models to video by incorporating texture-aware token merging and dual prompt tuning. The framework effectively mitigates temporal flickering and integrates multi-modal references, leading to an up to 0.74 dB PSNR gain and an 87.5% reduction in warping error across various tasks while reducing inference time by approximately two-thirds.
Key insights
- Dynamically adjusting token merging ratios based on local texture (smooth vs. texture-rich areas) is crucial for balancing temporal consistency with fine-grained detail preservation in video restoration.
- Optimizing prompt embeddings during both inversion and sampling phases of diffusion models can serve as an effective, training-free mechanism for reducing sampling steps and incorporating strong guidance for restoration tasks.
- Inflating 2D convolutional layers to 3D within pre-trained T2I LDMs, combined with targeted temporal consistency modules, can effectively overcome temporal flickering when adapting these models to video.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.