Reproduce “Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qianqian Wang, Ken Goldberg, Angjoo Kanazawa
The 'Robot See Robot Do' (RSRD) system from UC Berkeley enables robots to learn articulated object manipulation from a single human demonstration captured with a standard monocular camera. It achieves this by reconstructing and tracking the 3D motion of object parts, then allowing the robot to replicate the object's motion using its own movement strategies, demonstrating a 60% end-to-end success rate across various objects.
Key insights
- The object-centric approach bridges the morphological gap between human and robot manipulators by focusing on the desired object motion rather than the human's specific hand movements.
- 4D Differentiable Part Models (4D-DPM) accurately reconstruct 3D part motion from monocular video using differentiable rendering, DINO features, and As-Rigid-As-Possible (ARAP) regularization.
- Robots can achieve the same manipulation outcome as a human demonstration by planning their own bimanual trajectories and generating robot-specific grasps, not requiring direct copying of human actions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.