Reproduce “TextDeformer: Geometry Manipulation using Text Guidance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
William Gao, Noam Aigerman, Thibault Groueix, Vladimir G. Kim, Rana Hanocka
TextDeformer transforms existing triangle meshes toward text-described shapes while retaining source structure. It optimizes a Jacobian per triangle and reconstructs vertices globally, using rendered views, CLIP guidance, and view-consistency loss. On 111 prompt/source-mesh pairs, its CLIP-L/14 R-Precision is 55.2%, versus 55.4% for a vertex-displacement ablation, while self-intersections per face are 3.2% versus 67.7%. Retrieval scores are similar, but geometry favors Jacobians.
Key insights
- Optimizing per-triangle Jacobian matrices rather than direct vertex positions enables globally coherent and smooth mesh deformations, avoiding high-frequency noise and self-intersections.
- A novel view consistency loss ensures that semantic features from CLIP remain consistent for the same 3D point across different rendered viewpoints, preventing conflicting deformation signals.
- The method can simultaneously achieve both large-scale structural changes (e.g., elongating a neck) and add fine semantic details (e.g., animal spots) guided by text prompts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.