Reproduce “Dreamland: Controllable World Creation with Simulator and Generative Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sicheng Mo, Ziyang Leng, Leon Liu, Weizhen Wang, Honglin He, Bolei Zhou

UCLA researchers develop Dreamland, a hybrid framework that combines physics-based simulators with large-scale generative models for controllable world creation through a novel Layered World Abstraction (LWA) that decomposes scenes into traffic participants, map layouts, and background elements, achieving 52.3% better image quality (FID) and 17.9% improved depth alignment compared to SimGen while demonstrating zero-shot generalization across simulators and enabling precise object-level control for safety-critical scenario generation in autonomous driving, with their D3Sim dataset containing ~60,000 paired simulated and real-world driving conditions and downstream validation showing 3.9% accuracy improvements for embodied agents trained on Dreamland-generated synthetic data, alongside extensions to 4K video generation up to 121 frames that preserve both photorealistic quality and fine-grained simulator control through minimal adaptation of pre-trained foundation models.

Key insights

  • Stage-2 LWA-Sim2Real refinement is critical for bridging domain gaps, with ablation studies showing consistent improvements in both fidelity and controllability
  • Minimal adaptation of pre-trained generative models (fine-tuning only linear projection layers) preserves world knowledge while enabling structural control
  • The framework demonstrates excellent scalability, with performance gains when upgrading from SDXL to SD3 to Flux foundation models

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.