Reproduce “DiffHarmony: Latent Diffusion Model Meets Image Harmonization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pengfei Zhou, Fangxiang Feng, Xiaojie Wang
DiffHarmony adapts a pre-trained latent diffusion model for image harmonization, mitigating VAE-induced reconstruction errors through a two-stage process that includes higher-resolution inference and a dedicated refinement network. This approach achieves state-of-the-art quantitative results on the iHarmony4 dataset and produces visually superior harmonized images.
Key insights
- Performing harmonization at higher resolutions during inference (e.g., 1024px) significantly improves image quality metrics by reducing VAE information compression loss.
- The dedicated U-Net refinement stage effectively corrects residual image distortions, especially those arising from more aggressive VAE compression, demonstrating a complementary effect with high-resolution processing.
- The method successfully overcomes the VAE-induced reconstruction error, a major hurdle for applying pre-trained latent diffusion models to pixel-accurate image-to-image translation tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.