Reproduce “WonderJourney: Going from Anywhere to Everywhere” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T. Freeman, Forrester Cole, Deqing Sun and 3 more
WonderJourney presents a modular framework for perpetual 3D scene generation, synthesizing long, coherent sequences of diverse 3D environments from arbitrary text or image inputs, leading to visually compelling and user-preferred "wonderjourneys" that move beyond single-scene limitations. The system leverages state-of-the-art foundation models and achieves high visual quality and strong user preference over existing methods, as validated by qualitative examples and a human study.
Key insights
- Decomposing perpetual 3D scene generation into iterative steps (description, visual generation, validation) allows for effective leveraging of powerful, off-the-shelf foundation models.
- Strategic camera placement that creates empty space for outpainting is crucial for encouraging text-to-image models to generate new and diverse scene content, rather than simply extrapolating existing views.
- VLM-based visual validation serves as an effective quality control mechanism to automatically filter out undesired visual artifacts, significantly improving the consistency and overall quality of generated 'wonderjourneys'.
- Careful depth processing, including object boundary refinement and sky correction, is essential for accurately lifting 2D generated images into geometrically plausible 3D point clouds, which is vital for smooth scene transitions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.