Reproduce “Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jian Tang, Pu Pang, Haowen Sun, Chengzhong Ma, Xingyu Chen, Hua Huang, Xuguang Lan
Semantic 2D Gaussian Splatting (S2GS) offers a novel representation for robust sim-to-real policy transfer in robotic manipulation, extracting object-centric, domain-invariant spatial features. This approach, developed by researchers at Xi'an Jiaotong University and Shanghai Jiao Tong University, enabled a Diffusion Policy to achieve high success rates, such as 80-87%, across pick, push, and stack tasks in real-world environments with unseen objects.
Key insights
- Extracting object-centric, domain-invariant spatial features via S2GS can reduce the sim-to-real gap to a degree where policy performance approaches that of using ground-truth state inputs.
- 2D Gaussian primitives within S2GS offer superior appearance quality and geometric accuracy compared to 3D Gaussian Splatting, enhancing the precision of spatial feature extraction from object surfaces.
- Leveraging semantic understanding (via CLIP and SAM) within the representation allows for effective filtering of irrelevant background information, enabling the policy to focus on critical target objects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.