Reproduce “UrbanWorld: An Urban World Model for 3D City Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yu Shang, Yuming Lin, Yu Zheng, Hangyu Fan, Jingtao Ding, Jie Feng, Jiansheng Chen, Li Tian and 1 more

UrbanWorld introduces the first generative urban world model, capable of automatically creating realistic, customizable, and interactive 3D urban environments. The system, developed by Tsinghua University researchers, integrates an urban Multimodal Large Language Model with diffusion-based rendering to produce high-fidelity scenes that can support embodied AI agent learning and urban simulation, outperforming existing methods in realism, diversity, and interactivity.

Key insights

  • Integrating a specialized Multimodal Large Language Model (MLLM) with controllable diffusion-based rendering enables detailed urban scene design and high-quality texture generation.
  • An iterative 'map-design-render-refine' pipeline, including MLLM-assisted scene refinement, allows for high fidelity and consistency with user prompts.
  • An element-wise, UV-aware texture rendering strategy with position map-guided inpainting addresses challenges of texture mismatching and untextured areas in large 3D urban scenes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.