Reproduce “SHMT: Self-supervised Hierarchical Makeup Transfer via Latent Diffusion Models” on OpenResearch

About this paper

Zhaoyang Sun, Shengwu Xiong, Yaxiong Chen, Fei Du, Weihua Chen, Fan Wang, Yi Rong

A self-supervised hierarchical makeup transfer method, SHMT, utilizes latent diffusion models to decouple facial content and makeup representations for reconstruction, providing flexible control over source texture preservation versus reference makeup fidelity. This approach achieves superior performance, with SHMT-h4 reaching FID 27.01 and CLS 0.786 on LADN, and SHMT-h0 showing Key-sim 0.978, while also earning significant user preference.

Key insights

• Self-supervised learning within a decoupling-and-reconstruction framework can effectively replace the need for imperfect pseudo-paired data in makeup transfer, leading to higher fidelity results.
• Hierarchical representations of source texture details allow for adaptive control over content preservation, enabling the model to cater to varying makeup complexities from subtle enhancements to dramatic transformations.
• Dynamic, iterative alignment correction during the diffusion process, informed by the denoising progress, is crucial for refining semantic alignment and achieving superior makeup fidelity, especially for intricate styles.

Summary by alphaXiv.