Reproduce “UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziming Wang
Researchers at The University of Hong Kong and the University of Science and Technology of China developed UMI-3D, an enhanced robot demonstration interface that incorporates a lightweight LiDAR sensor for robust 3D spatial perception, overcoming vision-only limitations. This system allows for the reliable collection of data for complex manipulation tasks like curtain pulling and door opening, and policies trained on it show strong generalization across unseen object combinations and maintain compatibility with prior UMI systems.
Key insights
- Replacing monocular visual SLAM with LiDAR-centric SLAM on the robot's gripper profoundly enhances pose estimation robustness and 3D scene reconstruction reliability in challenging environments.
- Hardware-level synchronization and a unified spatiotemporal calibration framework are critical for fusing multimodal sensor data (LiDAR, camera, IMU) into consistent perception-action pairs.
- Representing actions as relative end-effector pose trajectories improves policy robustness by accounting for tracking errors and aligning with incremental SLAM state estimation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.