Reproduce “RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daniel Ho, Kanishka Rao, Zhuo Xu, Eric Jang, Mohi Khansari, Yunfei Bai

RetinaGAN introduces an object-aware approach to sim-to-real transfer, adapting simulated images to reality while preserving critical object semantics through a novel perception consistency loss. This method achieved 80% real-world grasping success, demonstrated over 10x data efficiency, and enabled training high-performance policies with minimal to no real-world task data.

Key insights

  • Explicitly enforcing object-level semantic consistency during sim-to-real image translation is crucial for preventing distortions of task-relevant information, making adapted data genuinely useful for robotic policies.
  • A pre-trained object detector, decoupled from the specific robotic task, can effectively guide the domain adaptation process, allowing for generalizable and reusable image translation across diverse robotic tasks and visual environments.
  • Balancing visual realism (via adversarial and cycle consistency losses) with semantic integrity (via perception consistency loss) through a weighted loss function is essential for achieving robust and reliable sim-to-real transfer.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.