Reproduce “Perception Stitching: Zero-Shot Perception Encoder Transfer for Visuomotor Robot Policies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pingcheng Jian, Easop Lee, Zachary Bell, Michael M. Zavlanos, Boyuan Chen
A framework called Perception Stitching facilitates the zero-shot transfer of visual perception encoders within visuomotor robot policies, enabling direct reuse of trained components across varied camera configurations. This approach attained success rates reaching 93% in complex simulated manipulation tasks and successfully transferred skills to real-world scenarios.
Key insights
- Latent spaces of independently trained perception encoders can exhibit misalignments, necessitating alignment strategies for successful module transfer.
- Encoding visual information as a "relative representation" against a set of anchor images helps achieve invariance to isometric transformations in the latent space, making modules reusable.
- Applying a disentanglement regularization loss during policy training encourages independent latent features, which is critical for robust zero-shot transfer across varied visual inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.