Reproduce “GHOST in the Robots: Real-Time Exocentric Dual-Robot VR Teleoperation from Onboard Cameras” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yichen Wei, Faisal Zaghloul, Soujanya C Aryal, Aanya K. Agrawal, Chengfan Li, Jason Xinyu Liu, James Tompkin, Stefanie Tellex
Researchers from Brown University and MIT developed GHOST, an open-source VR teleoperation system that enables a single human operator to precisely control two mobile manipulators using only onboard sensing. The system achieved up to 4 times higher success rates on certain novice tasks and a 1.47 times average speedup for experts compared to a tablet interface, while enabling new bimanual manipulation capabilities.
Key insights
- Real-time, exocentric 3D scene reconstruction from onboard robot cameras, enhanced by depth completion, significantly improves spatial awareness and task reliability for single-operator multi-robot coordination.
- Direct 6-DoF hand-pose mapping in VR, combined with a mode-switching control scheme, enables intuitive and precise bimanual manipulation and synchronized base control for two mobile manipulators.
- Predictive visual feedback through a virtual gripper target effectively mitigates perceived latency, enhancing operator confidence and control fluidity in teleoperation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.