Reproduce “R+X: Retrieval and Execution from Everyday Human Videos” on OpenResearch

About this paper

Georgios Papagiannis, Norman Di Palo, Pietro Vitiello, Edward Johns

Imperial College London's Robot Learning Lab developed R+X, a two-stage framework enabling robots to learn and execute skills from long, unlabelled first-person human videos using natural language commands. The system achieved an average success rate of 0.7 across 12 everyday tasks, demonstrating robust generalization to unseen objects, challenging spatial configurations, and complex language while preventing catastrophic forgetting.

Key insights

• Foundation models can be effectively leveraged as powerful tools for intelligent retrieval and in-context learning in robotics, circumventing the need for computationally expensive robot-specific fine-tuning.
• Representing human demonstrations through sparse 3D visual keypoints and stabilized hand joint actions facilitates robust cross-embodiment mapping and generalization across diverse environments and objects.
• A retrieval-augmented execution paradigm enables continuous, scalable skill acquisition from everyday human videos without suffering from catastrophic forgetting, a common challenge in sequential deep learning for robotics.

Summary by alphaXiv.