Reproduce “Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasets” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jeremiah Coholich, Justin Wit, Robert Azarcon, Zsolt Kira

MANGO, a sim2real image translation method, generates diverse, unseen real-world viewpoints from simulated data to enhance robot policy robustness. It employs a GAN-based architecture with novel loss functions, significantly improving policy success rates when camera viewpoints are shifted from the training distribution.

Key insights

  • Careful discriminator regularization, specifically using a highly-regularized PatchGAN, is essential to prevent image translation models from overfitting to fixed backgrounds in low-diversity robot datasets.
  • Modifying the InfoNCE loss to account for inherent content similarities in robot scenes, by reducing penalties for highly similar 'negative' samples, improves content preservation.
  • Leveraging ground-truth segmentation from simulation via a novel SegNCE loss effectively enhances semantic consistency and object boundary preservation during sim2real translation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.