Reproduce “Sensor Transfer: Learning Optimal Sensor Effect Image Augmentation for Sim-to-Real Domain Adaptation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexandra Carlson, Katherine A. Skinner, Ram Vasudevan, Matthew Johnson-Roberson

The Sensor Transfer Network learns and applies optimal physically-based sensor effects like chromatic aberration, blur, exposure, noise, and color balance from real-world datasets to synthetic images. This approach effectively reduces the sim-to-real domain gap, leading to improved object detection performance on real data, with gains of up to 5.35 AP50 for Cityscapes and 1.66 AP50 for KITTI datasets.

Key insights

  • Sensor-specific effects are a distinct and critical contributor to the sim-to-real domain gap, and modeling them explicitly is more effective than solely focusing on scene-level adaptations.
  • Learning optimal distributions of physically-based sensor effect parameters, rather than broad randomization or general image-to-image translation, yields superior domain adaptation and avoids introducing detrimental structural distortions.
  • The data-driven learning approach to sensor effect augmentation significantly improves data efficiency, enabling higher performance with smaller augmented synthetic datasets compared to manual domain randomization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.