Reproduce “MMSFormer: Multimodal Transformer for Material and Semantic Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md Kaykobad Reza, Ashley Prater-Bennette, M. Salman Asif
Researchers from UC Riverside and Air Force Research Laboratory developed MMSFormer, a multimodal transformer model featuring a novel fusion block that efficiently integrates an arbitrary number of diverse imaging modalities for material and semantic segmentation. The model achieved state-of-the-art performance on MCubeS, FMB, and PST900 datasets, demonstrating improved segmentation accuracy and contour precision while maintaining computational efficiency.
Key insights
- The Squeeze-and-Excitation (SE) channel attention block and parallel multi-scale convolutions are critical components of the fusion block, contributing significantly to feature calibration and multi-scale feature capturing, respectively.
- Model performance consistently improves as more modalities are incrementally integrated, validating the fusion block's ability to combine useful, complementary information from diverse inputs.
- Different modalities contribute uniquely to the identification of specific material classes (e.g., NIR aids asphalt/plastic, AoLP aids road markings/glass, DoLP aids plaster/sand).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.