Reproduce “Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings” on OpenResearch

About this paper

Kevin Chen, Christopher B. Choy, Manolis Savva, Angel X. Chang, Thomas Funkhouser, Silvio Savarese

Researchers from Stanford and Princeton developed a framework to learn joint embeddings for natural language descriptions and colored 3D shapes, enabling both text-to-shape retrieval and the novel task of text-to-colored-3D-shape generation. The system achieved 95.07% RR@1 for text-to-shape retrieval on a primitives dataset and generated realistic 3D shapes with an IoU of 9.64 and Earth Mover's Distance of 0.4443 for color matching, surpassing existing baselines.

Key insights

• Instance-level association learning, combined with multimodal metric learning, effectively establishes cross-modal links between freeform natural language and complex 3D shapes without requiring fine-grained category or attribute labels.
• A Conditional Wasserstein GAN (CWGAN) can successfully address the challenges of generating high-quality, diverse, and text-conditioned colored 3D voxel models, overcoming limitations of previous generative models in the 3D domain.
• The learned joint embedding space encodes rich semantic information, where specific dimensions correlate with distinct shape attributes (e.g., color, geometry), enabling semantic vector arithmetic for manipulating generated shapes.

Summary by alphaXiv.