Reproduce “Zero-shot Inexact CAD Model Alignment from a Single Image” on OpenResearch

About this paper

Pattaramanee Arsomngern, Sasikarn Khwanmuang, Matthias Nießner, Supasorn Suwajanakorn

Researchers at VISTEC and Technical University of Munich developed a weakly-supervised method for zero-shot 9-DoF alignment of inexact 3D CAD models from a single image, which injects geometry awareness into foundation model features and utilizes texture-invariant dense optimization. The approach achieved state-of-the-art zero-shot generalization on a newly introduced SUN2CAD dataset and outperformed existing supervised methods on seen categories.

Key insights

• Injecting geometry and part-symmetry awareness into foundation model features (e.g., DINOv2) significantly enhances their utility for precise 3D alignment, overcoming issues like texture sensitivity and symmetry ambiguities.
• Performing dense pose refinement in Normalized Object Coordinate (NOC) space, along with silhouette and depth spaces, enables robust alignment of inexact CAD models by being invariant to texture differences between the image and the model.
• A weakly-supervised training paradigm, utilizing only canonically aligned CAD models and synthetic renderings, can achieve state-of-the-art zero-shot generalization for 9-DoF inexact CAD model alignment, mitigating the need for costly scene-level pose annotations.

Summary by alphaXiv.