Reproduce “CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, Rainer Stiefelhagen

The CMX framework generalizes across diverse complementary sensor modalities, including depth, thermal, polarization, event, and LiDAR, for RGB-X semantic segmentation. It consistently achieves state-of-the-art performance on numerous benchmarks, such as 56.9% mIoU on NYU Depth V2 and 59.7% on MFNet, demonstrating mIoU improvements up to +6.1% over RGB-only baselines.

Key insights

  • A unified, modality-agnostic Transformer-based framework can effectively generalize across a wide range of RGB-X modalities, overcoming the limitations of previous modality-specific designs.
  • Comprehensive cross-modal interaction mechanisms, including feature rectification and efficient cross-attention, are critical for adaptively handling varying sensor characteristics and enhancing feature quality.
  • Vision Transformers, with their ability to capture long-range dependencies, provide a robust architectural foundation for complex multi-modal fusion tasks, outperforming traditional CNN-based approaches in efficiency and accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.