Reproduce “Vision-Language Model Guided Image Restoration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cuixin Yang, Rongkang Dong, Kin-Man Lam
VLMIR, a two-stage framework from The Hong Kong Polytechnic University, integrates comprehensive visual and linguistic priors from Vision-Language Models to improve image restoration. The method demonstrates robust performance in universal tasks such as raindrop removal, dehazing, and denoising, yielding higher fidelity and perceptual quality compared to prior approaches.
Key insights
- Explicitly leveraging both visual (clean content, degradation) and refined linguistic (caption) priors from VLMs at a feature level enhances semantic coherence and visual quality in image restoration.
- Aligning noisy low-quality image captions with accurate high-quality ground truth captions through LoRA fine-tuning improves the reliability and utility of linguistic guidance.
- Combining cross-attention mechanisms for semantic and image content embeddings with direct degradation injection effectively conditions generative diffusion models for robust universal restoration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.