Reproduce “Mirage: One-Step Video Diffusion for Photorealistic and Coherent Asset Editing in Driving Scenes” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shuyun Wang, Haiyang Sun, Bing Wang, Hangjun Ye, Xin Yu

Mirage, a system developed by researchers from The University of Queensland and Xiaomi EV, presents a one-step video diffusion model for photorealistic and temporally coherent asset editing in driving scenes. It enhances spatial fidelity and temporal consistency by outperforming prior methods like R3D2, achieving a PSNR of 28.60, SSIM of 0.914, LPIPS of 0.069, and vFID of 0.178, while also providing a high-quality dataset for autonomous driving data augmentation.

Key insights

  • Spatial detail loss in causal 3D VAEs can be effectively mitigated by injecting temporally agnostic high-frequency 2D encoder latents into the decoder's final upsampling blocks without compromising temporal causality.
  • Precise data alignment for synthetic asset insertion requires both 3D coarse matching and 2D fine refinement to bridge the distribution gap between scene and asset representations, yielding clean supervisory signals.
  • A two-stage training approach, first adapting the VAE for reconstruction fidelity and then harmonizing for appearance quality with specialized LoRA layers, leads to superior photorealistic and temporally coherent video editing.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.