Reproduce “MC-VTON: Minimal Control Virtual Try-On Diffusion Transformer” on OpenResearch

About this paper

Junsheng Luan, Guangyuan Li, Lei Zhao, Wei Xing

MC-VTON by Zhejiang University presents a Diffusion Transformer approach for virtual try-on that achieves high fidelity with minimal control. It eliminates the need for extra networks or complex inputs by directly integrating conditions into the DiT's backbone and utilizes parameter-efficient LoRAs to train with only 1.03% of the backbone's parameters. This method generates high-quality 1024x768 try-on images in just 8 inference steps, outperforming existing diffusion models in quality and efficiency on datasets like VITON-HD and DressCode.

Key insights

• The intrinsic multi-modal attention mechanisms of Diffusion Transformers (DiT) can effectively integrate conditional information (garment, masked person) without needing additional reference networks or image encoders, simplifying architecture and input requirements.
• Parameter-efficient LoRA (Low-Rank Adaptation) can be refined with task-specific LoRAs (G-LoRA for garment, MP-LoRA for masked person) and dynamic switches to efficiently learn distinct feature interactions for complex, multi-faceted tasks like virtual try-on.
• Adversarial distillation diffusion applied in latent space enables significant reduction in inference steps for DiT-based VTON models, allowing high-quality image generation in as few as 8 steps without compromising fidelity.

Summary by alphaXiv.