Reproduce “FetchBot: Learning Generalizable Object Fetching in Cluttered Scenes via Zero-Shot Sim2Real” on OpenResearch

About this paper

Weiheng Liu, Yuxuan Wan, Jilong Wang, Yuxuan Kuang, Wenbo Cui, Xuesong Shi, Haoran Li, Dongbin Zhao and 2 more

FetchBot introduces a framework for robots to perform generalizable object fetching in highly cluttered scenes via zero-shot sim-to-real transfer. The system integrates a voxel-based scene generator, a dynamics-aware oracle policy for data collection, and a vision-based policy leveraging foundation models for consistent depth prediction and learned 3D semantic occupancy, achieving real-world success rates up to 93.3% while minimizing environmental disturbance.

Key insights

• Inferring complete 3D semantic occupancy, even for occluded regions, is crucial for robust and safe object fetching in highly cluttered environments, outperforming methods relying on partial observations.
• Leveraging a vision foundation model like DepthAnything to predict depth from RGB images effectively bridges the sim-to-real gap for visual input, handling real-world sensor noise and challenging materials.
• Generating large-scale, densely cluttered synthetic scenes with accurate ground truth occupancy and expert demonstrations significantly improves the generalization and robustness of learned fetching policies.
• A compact, task-appropriate Region of Interest (ROI) for 3D feature processing enhances policy generalization and accuracy by focusing on local context relevant to manipulation.

Summary by alphaXiv.