Reproduce “Painterly Image Harmonization using Diffusion Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lingxiao Lu, Jiangtong Li, Junyan Cao, Li Niu, Liqing Zhang
Researchers from Shanghai Jiao Tong University introduce PHDiffusion, a novel framework leveraging a Stable Diffusion model for painterly image harmonization. It effectively integrates photographic objects into paintings by providing guided style transfer and content preservation, yielding visually coherent results preferred by users over previous methods.
Key insights
- Leveraging a frozen pre-trained Stable Diffusion model provides a strong generative prior, reducing training burden while focusing learning on harmonization-specific guidance.
- A resolution-dependent fusion strategy within the Dual Encoder Fusion (DEF) module is effective for integrating multi-scale content and style information into the denoising process.
- A specific combination of noise, AdaIN style, contrastive style, and content losses is essential to achieve a controlled balance between transferring artistic style and preserving foreground content.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.