Reproduce “RwoR: Generating Robot Demonstrations from Human Hand Collection for Policy Learning without Robot” on OpenResearch

About this paper

Liang Heng, Xiaoqi Li, Shangqing Mao, Jiaming Liu, Ruolin Liu, Jingli Wei, Yu-Kai Wang, Yueru Jia and 4 more

RwoR introduces an end-to-end system for robot imitation learning that converts human hand demonstrations captured by a wrist-mounted camera into robot-ready demonstrations using a diffusion-based generative model. This approach effectively bridges the visual domain gap, enabling policies trained on these generated demonstrations to achieve an average success rate of 0.78, comparable to specialized UMI gripper data at 0.82, without requiring a physical robot for data collection.

Key insights

• A data-driven, diffusion-based generative model can effectively and robustly bridge the visual domain gap between human hands and robot grippers, outperforming rule-based approaches.
• Meticulous data pre-processing, including temporal alignment and foreground-background segmentation, is crucial for training a generative model that accurately transforms only the relevant foreground elements while preserving background consistency.
• High-quality synthetic robot demonstrations can enable visuomotor policies to achieve performance comparable to those trained on real robot data, even generalizing to unseen manipulation actions and object instances.

Summary by alphaXiv.