Reproduce “Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yash Yardi, Samuel Biruduganti, Lars Ankile

The authors present an offline framework for screening vision encoders. It measures whether frozen image features support action prediction and align across simulated and real scenes, evaluating 23 encoders. Manipulation-pretrained models tend to score higher on action recovery, while CNNs generally score higher on domain alignment; MCR ranks near the top on both, but transfer prediction remains unvalidated.

Key insights

  • The relevance of pre-training data is paramount for effective Sim2Real transfer, with manipulation-specific datasets consistently yielding better performance than general-purpose ones.
  • CNN architectures generally exhibit stronger domain invariance and robustness to Sim2Real distribution shifts compared to Vision Transformers, indicating architectural biases can play a crucial role.
  • Model capacity (parameter count) does not directly correlate with improved Sim2Real performance, suggesting that the quality and relevance of pre-training are more important than sheer scale.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.