Reproduce “DriveCtrl: Conditioned Sim-to-Real Driving Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haonan Zhao, Yiting Wang, Jingkun Chen, Valentina Donzella, Thomas Bashford-Rogers, Kurt Debattista
DriveCtrl, developed by researchers at the University of Warwick and collaborators, presents a depth-conditioned framework for generating realistic driving videos from simulated data while preserving frame-level annotations. The framework produced videos that enhanced real-world semantic segmentation performance and was evaluated using a new driving-domain-specific realism score.
Key insights
- Conditioning a pretrained video foundation model with structural depth maps is effective for sim-to-real driving video translation, preserving crucial scene layout and motion patterns.
- Decoupling the visual style guidance from structural generation using random, unpaired reference images ensures that the model maintains geometric fidelity dictated by the depth input.
- A world-knowledge-informed, driving-domain-specific evaluation metric (DVRS) is essential for comprehensively assessing the realism, plausibility, and utility of generated driving videos.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.