Reproduce “ShapeClipper: Scalable 3D Shape Learning from Single-View Images via Geometric and CLIP-based Consistency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zixuan Huang, Varun Jampani, Anh Thai, Yuanzhen Li, Stefan Stojanov, James M. Rehg
ShapeClipper learns high-quality 3D object shapes from single-view segmented images by integrating CLIP-based semantic consistency and geometric consistency from off-the-shelf normal estimators. This approach bypasses expensive 3D or multi-view supervision, achieving superior reconstruction quality on challenging real-world datasets like Pix3D and Pascal3D+.
Key insights
- CLIP's robust semantic embeddings, which are somewhat viewpoint-invariant, can effectively serve as a powerful signal for inferring global shape properties and generating implicit multi-view supervision from single images.
- Off-the-shelf surface normal estimators, despite their inherent noise for object-centric images, can be pragmatically leveraged with a noise-tolerant optimization strategy to inject crucial local geometric details into the learning process.
- Combining these two complementary consistency constraints—CLIP-based semantic for global shape and geometric for local details—effectively addresses the dual challenges of top-down and bottom-up reasoning in single-view 3D reconstruction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.