Reproduce “Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks” on OpenResearch

About this paper

Xinchen Yan, Mohi Khansari, Jasmine Hsu, Yuanzheng Gong, Yunfei Bai, Sören Pirk, Honglak Lee

Researchers developed a two-step framework for data-efficient sim-to-real robotic grasping, which achieved a 61% real-world grasping success rate on unseen objects without requiring any real-world grasping data. The framework leverages a self-supervised deep point cloud prediction network to learn domain-invariant 3D object representations, outperforming a 2.5D baseline by 10 percentage points in real-world performance.

Key insights

• Learning a compact, domain-invariant 3D shape representation (point cloud) serves as a robust intermediate interface for sim-to-real transfer, less sensitive to real-world visual variations and sensor noise.
• Multi-view self-supervision provides an effective mechanism to train deep networks for full 3D point cloud reconstruction from real-world RGBD images without requiring expensive 3D annotations.
• Combining a robust 3D perception module with a grasping policy trained exclusively in simulation can significantly reduce the data collection bottleneck for real-world robotic manipulation tasks.

Summary by alphaXiv.