Reproduce “MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping” on OpenResearch

About this paper

Haofei Lu, Hongjia Liu, Yifei Dong, Florian T. Pokorny, Jens Lundell, Danica Kragic

MoDex introduces the first diffusion policy for sequential multi-object dexterous grasping, enabling a single hand to pick and securely hold multiple items without releasing previously grasped ones. The policy achieves up to 75% success in simulation and demonstrates promising zero-shot real-world transfer by strategically utilizing finger DoFs and leveraging a two-stage imitation and reinforcement learning paradigm.

Key insights

• Strategically utilizing a subset of fingers for each grasp, guided by explicit OS and grasp history conditioning, allows dexterous hands to effectively perform sequential multi-object manipulation.
• A hybrid training approach, combining expert demonstrations via IL with fine-tuning through DPPO and a dense reward function, significantly enhances policy robustness and sim-to-real transfer for complex manipulation tasks.
• Directly mapping raw perceptual inputs to continuous actions via diffusion policies provides a generalized and adaptable solution for multi-object grasping, reducing reliance on explicit planning and object models.

Summary by alphaXiv.