Reproduce “LaST-HD: Learning Latent Physical Reasoning from Scalable Human Data for Robot Manipulation” on OpenResearch

About this paper

Jiaming Liu, Yinxi Wang, Chenyang Gu, Siyuan Qian, Xiangju Mi, Hao Chen, Jiawei Chen, Qingpo Wuwu and 10 more

LaST-HD, a framework from Peking University and collaborators, enables robots to learn manipulation skills from scalable human-hand demonstrations by aligning human and robot actions within a shared latent physical reasoning space. This approach, supported by a novel data collection glove and progressive training, improves generalization to unseen scenarios and allows for rapid adaptation with minimal human input.

Key insights

• Aligning human and robot actions in a latent physical reasoning space, which captures the physical consequences of interactions rather than just kinematics, is crucial for robust cross-embodiment transfer.
• High-fidelity human-hand data, collected efficiently with the OOL Glove, can effectively supplement or even outperform equivalent robot data when correctly aligned, demonstrating efficient transfer of interaction priors.
• A progressive training approach that includes online human-hand correction using the OOL Glove allows for rapid adaptation and policy refinement in novel or failure-prone scenarios with minimal additional data.

Summary by alphaXiv.