Reproduce “DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image” on OpenResearch

About this paper

Daoyi Gao, Dávid Rozenberszki, Stefan Leutenegger, Angela Dai

DiffCAD, from Technical University of Munich, presents a weakly-supervised probabilistic method for CAD model retrieval and alignment from a single RGB image, utilizing disentangled diffusion models. The approach, trained solely on synthetic data, outperforms fully-supervised methods like SPARC and ROCA on the ScanNet benchmark, achieving 32.1% alignment accuracy and 0.075 Chamfer Distance with 8 hypotheses.

Key insights

• Weak supervision, by training solely on synthetic data and using machine-generated 2D estimates for real-world inference, can achieve performance superior to fully-supervised methods for CAD retrieval and alignment.
• Probabilistic modeling with disentangled diffusion models effectively captures and addresses inherent ambiguities in monocular 3D perception, allowing for the generation of multiple plausible hypotheses (e.g., for scale, pose, and shape).
• Disentangling the prediction of scene scale, object pose via Normalized Object Coordinates (NOCs), and object shape latent codes improves robustness and generalization, especially when adapting from synthetic to real-world data.

Summary by alphaXiv.