Reproduce “ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanwen Zou, Chenyang Shi, Wenye Yu, Han Xue, Jun Lv, Ye Pan, Chuan Wen, Cewu Lu
ActiveGlasses trains robots from bare-hand demonstrations recorded with smart glasses, predicting both an object’s 3D trajectory and camera movement. A robot uses an arm to move the camera while manipulating the object, without retargeting human hand motion. In book placement, it achieved 14 final-stage successes in 20 trials, versus 7 without active vision; these results are limited to the tested tasks.
Key insights
- Explicitly incorporating human 6-DoF head movements as a learnable active vision signal significantly improves robot performance in tasks with occlusion and precision requirements.
- An object-centric 3D point-cloud representation enables robust policy learning and direct zero-shot transfer to various robot platforms by abstracting away embodiment-specific details.
- Predicting absolute object trajectories and relative head movements optimizes policy learning, preventing overfitting and addressing robot workspace limitations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.