Reproduce “Visuomotor Grasping with World Models for Surgical Robots” on OpenResearch

About this paper

Hongbin Lin, Bin Li, Kwok Wai Samuel Au

Researchers at the Chinese University of Hong Kong developed GASv2, a visuomotor learning framework for surgical robots that enables object-agnostic grasping through sim-to-real transfer using only a standard endoscopic stereo camera. This framework achieved a 75% success rate on real dVRK robots, demonstrating robust generalization to diverse objects and conditions, including in *ex vivo* animal studies, significantly outperforming prior baselines.

Key insights

• A specialized perception pipeline tailored for surgical scenes, incorporating explicit depth, object segmentation, and scale-equivariant image representations like DSA, is crucial for effective visuomotor control with low-resolution inputs.
• Integrating deep reinforcement learning with traditional control components, such as PID and a 'Virtual Clutch', significantly enhances learning stability, improves safety, and facilitates successful sim-to-real transfer for complex manipulation tasks.
• Extensive domain randomization in simulation, applied to camera poses, object properties, and environmental noise, is vital for training object-agnostic policies that generalize robustly to diverse and unseen real-world surgical conditions.

Summary by alphaXiv.