Reproduce “RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks” on OpenResearch

About this paper

Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, Jiajun Wu

RoboCraft shapes Play-Doh toward target forms with a two-finger gripper. It represents the material as particles, learns their interactions with a graph neural network from particle distributions rather than point identities, and plans pinches using the learned model. The authors report shaping toward unused goals after ten minutes of random interaction data, though letter A was only roughly achieved.

Key insights

• A robust particle-based perception pipeline, augmented with physics and shape priors, can effectively extract meaningful state representations of deformable objects from raw RGBD data, even under severe occlusion.
• Graph Neural Networks, combined with distribution-based loss functions like Earth Mover's Distance and Chamfer Distance, can learn complex elasto-plastic dynamics from visual data without requiring explicit particle-to-particle correspondences.
• Leveraging a learned differentiable dynamics model within a gradient-based Model Predictive Control (MPC) framework enables precise and generalizable manipulation of high-DoF deformable objects with remarkable data efficiency.

Summary by alphaXiv.