Reproduce “SPIN: Simultaneous Perception, Interaction and Navigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shagun Uppal, Ananye Agarwal, Haoyu Xiong, Kenneth Shaw, Deepak Pathak
The SPIN framework enables mobile manipulators to reactively navigate and interact in complex, dynamic environments by integrating perception, interaction, and navigation into a single end-to-end policy. It employs an active visual system and a two-phase training strategy to achieve higher success rates in real-world scenarios compared to classical methods, demonstrating emergent behaviors like dynamic obstacle avoidance and whole-body coordination.
Key insights
- The tight coupling between perception and motion is essential for effective mobile manipulation, allowing the robot to actively "move to see and see to move."
- Active vision with an actuated camera significantly outperforms fixed-camera systems, demonstrating its necessity for effective navigation in cluttered environments.
- A two-phase training strategy, starting with reinforcement learning on simplified "scandot" representations and then distilling to depth images, provides a computationally tractable path for end-to-end visuomotor policy learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.