Reproduce “3DP3: 3D Scene Perception via Probabilistic Programming” on OpenResearch

About this paper

Nishad Gothoskar, Marco Cusumano-Towner, Ben Zinberg, Matin Ghavamizadeh, Falk Pollok, Austin Garrett, Joshua B. Tenenbaum, Dan Gutfreund and 1 more

The 3DP3 framework from MIT and the MIT-IBM Watson AI Lab addresses robust 3D scene understanding by integrating explicit 3D modeling with probabilistic programming. This approach accurately determines 6DoF object poses and hierarchical scene structure from RGB-D images, outperforming deep learning baselines in challenging conditions such as occlusion and stacked objects on the YCB-Video dataset.

Key insights

• Explicitly modeling 3D scenes with hierarchical scene graphs and physical contact relationships significantly enhances scene understanding and object pose estimation, particularly in challenging scenarios.
• Probabilistic programming provides a flexible framework for defining structured generative models and performing robust Bayesian inference, allowing for reasoning about uncertainty and incorporating prior knowledge about physical constraints.
• A hybrid inference approach combining fast bottom-up pose proposals with refined MCMC sampling, including novel involutive kernels for scene graph exploration, can effectively tackle complex 3D perception problems.

Summary by alphaXiv.