Reproduce “RealMaster: Lifting Rendered Scenes into Photorealistic Video” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dana Cohen-Bar, Ido Sobol, Raphael Bensadoun, Shelly Sheynin, Oran Gafni, Or Patashnik, Daniel Cohen-Or, Amit Zohar
RealMaster converts rendered 3D engine videos into photorealistic sequences, precisely preserving underlying scene structure, motion, and character identity. The framework leverages a two-stage approach and a fine-tuned video diffusion model, achieving superior realism and fidelity compared to existing video editing methods.
Key insights
- Video diffusion models can function as a "learned second-stage renderer," combining the precise control of 3D engines with the photorealism capabilities of generative AI without compromising structural integrity.
- Edge maps serve as critical structural cues for maintaining object boundaries, motion, and identity during the sparse-to-dense propagation in the paired data generation pipeline.
- A dedicated model trained on carefully constructed paired data offers improved generalization and avoids the limitations of directly running the multi-step data generation pipeline for inference, particularly for mid-sequence object appearances.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.