Reproduce “UMSS: Towards Unsupervised Multi-modal Semantic Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haitian Zhang, Thai Duy Nguyen, Xiangyuan Wang, Mohan Liu, Lin Wang
A framework for Unsupervised Multi-modal Semantic Segmentation (UMSS), named UniM2, addresses the challenge of integrating heterogeneous sensor data without annotations by transforming "fusion degradation" into consistent performance gains. It achieved mIoU improvements of 6.4% on NYU-Depth-v2 and 9.8% on MFNet over existing RGB-only unsupervised baselines.
Key insights
- Naive multi-modal fusion in unsupervised learning often degrades performance due to unaddressed inter-modal conflicts, rather than a lack of complementary information.
- Intrinsic label-free supervision, derived from consistent structural correspondences across modalities, can effectively guide the learning of a unified semantic space.
- Mitigating conflicts through an asymmetrical harmonization mechanism, anchoring to a reliable primary modality like RGB, is essential for successful unsupervised multi-modal integration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.