Reproduce “Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rishabh Jangir, Nicklas Hansen, Sambaran Ghosal, Mohit Jain, Xiaolong Wang
Researchers at UCSD developed "Look Closer," a multi-view vision system for robotic manipulation that integrates egocentric and third-person camera feeds using a Transformer with cross-view attention. This system enabled robots to perform precision tasks from raw, uncalibrated visual feedback, achieving a 75% success rate for hammering and 80% for pushing on a real robot, significantly improving Sim2Real transfer.
Key insights
- Intelligently fusing complementary egocentric (local, active) and third-person (global context) camera views provides a richer perceptual input necessary for precise robotic manipulation.
- A Transformer with bidirectional cross-view attention effectively learns spatial relationships and aggregates information across disparate views, leading to a more robust joint representation than simple feature concatenation.
- Learning directly from raw, uncalibrated multi-view images, combined with specific architectural design and data augmentation, can significantly bridge the Sim2Real gap and reduce the engineering overhead in deploying robotic policies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.