Reproduce “CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yankai Fu, Qiuxuan Feng, Ning Chen, Zichen Zhou, Mengzhen Liu, Mingdong Wu, Tianxing Chen, Shanyu Rong and 3 more

Peking University researchers developed CordViP, a visuomotor policy for dexterous robotic manipulation that addresses challenges in 3D perception by generating robust, 'interaction-aware' point clouds using 6D pose estimation and digital twins. The policy explicitly learns fine-grained contact and coordination correspondences between the hand and objects, achieving over 85% success rates on various real-world manipulation tasks while demonstrating superior data efficiency and generalization to unseen objects and varying conditions.

Key insights

  • Learning explicit spatial and temporal correspondences, such as contact maps between the hand and object and coordination between the hand and arm, is crucial for robust dexterous manipulation.
  • Robust 3D object pose estimation from a single view, combined with digital twin technology and robot kinematics, can generate high-quality, 'interaction-aware' point clouds that are complete and resilient to occlusions.
  • Pre-training an encoder with tasks focused on contact prediction and hand-arm coordination enhances the learned features, making the subsequent diffusion policy more effective and sample-efficient for complex manipulation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.