Reproduce “A Universal Semantic-Geometric Representation for Robotic Manipulation” on OpenResearch

About this paper

Tong Zhang, Yingdong Hu, Hanchen Cui, Hang Zhao, Yang Gao

Researchers at Tsinghua University and Shanghai AI Lab developed the Semantic-Geometric Representation (SGR), a hybrid perception module that integrates high-level semantic understanding from pre-trained 2D vision models with low-level 3D spatial reasoning from point clouds. This system achieved an average 68.6% success rate in simulation, a 1.43x improvement over baselines, and demonstrated enhanced generalization to novel object attributes in both simulated and real-robot manipulation tasks.

Key insights

• The synergistic integration of pre-trained 2D semantics with efficient 3D geometry in SGR provides a more robust and generalizable perception module for robotic manipulation than unimodal or conventional fusion approaches.
• Leveraging the broad semantic knowledge from large-scale pre-trained 2D models (e.g., CLIP) is crucial for enabling robots to generalize to novel semantic attributes like unseen colors and shapes.
• Directly adding raw RGB colors to 3D point cloud inputs in the geometric branch can reduce performance, highlighting the importance of structured, pre-trained semantic encoding and fusion rather than simple concatenation.

Summary by alphaXiv.