Reproduce “Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiaming Zhang, Kailun Yang, Chaoxiang Ma, Simon Reiß, Kunyu Peng, Rainer Stiefelhagen

Researchers from Karlsruhe Institute of Technology and ByteDance introduce Trans4PASS, a Transformer architecture with Deformable Patch Embedding and Deformable MLP that intrinsically adapts to distortions in panoramic images. They also propose Mutual Prototypical Adaptation (MPA) for unsupervised domain adaptation from pinhole to panoramic data, improving mIoU on DensePASS by 14.39% and achieving near fully-supervised performance on Stanford2D3D-Panoramic.

Key insights

  • Explicitly modeling panoramic distortions through deformable components (DPE, DMLP) within a Transformer architecture significantly improves semantic segmentation accuracy by better aligning with distorted object shapes.
  • Aligning semantic prototypes across source and target domains provides an effective mechanism for unsupervised domain adaptation, bridging the PIN2PAN gap without relying on target domain labels.
  • Combining architectural innovations for distortion awareness with a robust domain adaptation strategy allows models to achieve high performance on panoramic images, even when trained primarily on pinhole data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.