Reproduce “MV-UMI: A Scalable Multi-View Interface for Cross-Embodiment Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Omar Rayyan, John Abanes, Mahmoud Hafez, Anthony Tzes, Fares Abu-Dakka

The MV-UMI framework from New York University Abu Dhabi enhances robot manipulation data collection by integrating egocentric and processed third-person camera views to facilitate cross-embodiment learning. This approach improves policy performance by approximately 47% in sub-tasks requiring broad scene understanding, effectively mitigating domain shift through real-time human masking and background inpainting during data processing.

Key insights

  • Removing the human demonstrator from the third-person view during training is critical for preventing policies from overfitting to embodiment-specific cues, which is essential for effective cross-embodiment transfer.
  • Augmenting egocentric views with a processed, contextual third-person perspective significantly enhances policy performance in tasks demanding broader scene understanding and long-horizon planning.
  • Policies trained on human-masked data can maintain robustness even when the robot is present (unsegmented) in the third-person view during inference, suggesting simplified deployment architectures are possible.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.