Reproduce “Multi-View Masked World Models for Visual Robotic Manipulation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Younggyo Seo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, Pieter Abbeel

Multi-View Masked World Models (MV-MWM) introduces a framework for visual robotic manipulation that leverages self-supervised learning from multi-view video data. It achieves robust zero-shot sim-to-real transfer, enabling robots to perform tasks with uncalibrated and even hand-held cameras.

Key insights

  • Explicitly masking entire viewpoints during self-supervised learning ('view-masking') is crucial for forcing the model to learn synergistic, cross-view information for robust perception.
  • Training with strong viewpoint randomization in simulation allows policies to acquire sufficient robustness for zero-shot transfer to real robots, even with uncalibrated and dynamic camera setups.
  • In-domain, continually updated self-supervised representation learning from multi-view data can outperform general-purpose pre-trained vision models for specific visual robot control tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.