Reproduce “OCRA: Object-Centric Learning with 3D and Tactile Priors for Human-to-Robot Action Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kuanning Wang, Ke Fan, Yuqian Fu, Siyu Lin, Hu Luo, Daniel Seita, Yanwei Fu, Yu-Gang Jiang and 1 more
Researchers at Fudan University and the University of Southern California developed OCRA, an object-centric framework that enables robots to learn complex manipulation skills by observing human demonstration videos. It achieves robust action transfer by integrating multi-view 3D scene reconstruction and tactile sensing, leading to high success rates across various vision-only and visuo-tactile tasks.
Key insights
- Object-centric 3D representations derived from multi-view video reconstruction, rather than raw depth maps, are critical for robust generalization in robot manipulation tasks.
- Integrating pre-trained tactile sensing enables robots to perceive non-visual object properties, significantly expanding the range of manipulable objects and task complexities.
- The ResFiLM fusion module effectively combines visual and tactile features, allowing tactile signals to act as adaptive residual corrections for stable training and robust multimodal perception.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.