Reproduce “Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow” on OpenResearch

About this paper

Quan Meng, Yujin Chen, Lei Li, Matthias Nießner, Angela Dai

Seen2Scene introduces a 3D generative model that completes partial real-world scene scans and generates novel scenes by directly training on incomplete, noisy real-world data. This approach leverages visibility-guided flow matching, achieving improved 3D scan completion performance with a Chamfer Distance of 2.05x10⁻² and L2 of 0.51x10⁻⁴, and generating scenes with superior perceptual quality, indicated by a DINOv2-FID of 1.90x10².

Key insights

• Explicitly masking out unobserved regions during training (visibility-guided approach) is crucial for the model to learn a complete latent representation from partial inputs and avoid ambiguity.
• Directly training on a mix of real-world and synthetic 3D scan data significantly improves generalization, enabling the model to handle complex, realistic layouts better than synthetic-only training.
• Using open-vocabulary semantic embeddings (CLIP) for object labels and augmenting these with LLM-generated synonyms enhances robustness and flexibility for diverse object descriptions and text-to-3D generation.

Summary by alphaXiv.