Reproduce “Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations” on OpenResearch

About this paper

Irmak Guzey, Haozhi Qi, Julen Urain, Changhao Wang, Jessica Yin, Krishna Bodduluri, Mike Lambeta, Lerrel Pinto and 5 more

AINA is a framework enabling multi-fingered robots to learn complex manipulation skills from human demonstrations captured in everyday environments using smart glasses. The system directly translates human 3D hand and object movements into robot actions, successfully performing tasks like pushing and picking with an average 86% success rate across multiple tasks, without requiring any robot-specific training data.

Key insights

• Smart glasses provide sufficiently rich and scalable egocentric 3D data (hand poses, stereo depth, object point clouds) that can effectively bridge the human-robot embodiment gap for dexterous manipulation tasks.
• Combining diverse 'in-the-wild' human demonstrations with a single 'in-scene' demonstration is crucial for robust spatial and orientational alignment, enabling successful generalization and anchoring of policies.
• 3D point-based policies are significantly more robust to visual disparities and viewpoint changes between human demonstrations and robot deployment than image-based methods, underscoring the importance of 3D geometric inputs for human-to-robot transfer.

Summary by alphaXiv.