Reproduce “PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cheng Qian, Julen Urain, Kevin Zakka, Jan Peters
PianoMime enables a robotic agent to learn dexterous piano playing from internet videos by converting 2D human demonstrations into 3D robot-actionable data. The system combines imitation and reinforcement learning to train song-specific policies, then distills this knowledge into a generalist policy, demonstrating the ability to play hundreds of different songs in a MuJoCo simulation environment.
Key insights
- Internet videos, even without specialized motion capture, can be effectively transformed into usable 3D robot demonstration data for complex dexterous tasks by combining computer vision with external contextual information like MIDI files.
- A hybrid learning approach combining imitation learning (mimicking human trajectories) with reinforcement learning (optimizing for task success) and residual policy learning leads to improved dexterous performance.
- A hierarchical policy architecture, coupled with learned representations from Signed Distance Fields (SDFs), enables a generalist robotic policy to effectively generalize complex skills to new, unseen scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.