Reproduce “MPerS: Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ziyi Wang, Xianping Ma, Ziyao Wang, Hongyang Zhang, Man On Pun

Researchers at The Chinese University of Hong Kong (Shenzhen) developed MPerS, a framework for remote sensing scene segmentation that generates high-quality, multi-perspective captions from multiple MLLMs and dynamically integrates them into visual features. This method achieved state-of-the-art performance, with 78.42% mIoU on Vaihingen, 81.04% mIoU on Potsdam, and 72.17% mIoU on SynDrone, particularly enhancing the segmentation of complex and dense small objects.

Key insights

  • Generating high-quality, multi-perspective captions from MLLMs, covering land-cover types, proportions, and spatial relationships, is crucial for accurate RS scene perception.
  • Dynamically integrating semantics from multiple MLLM experts provides a more comprehensive and robust textual guide than single-expert approaches.
  • The Linguistic Query Guided Attention mechanism effectively fuses textual and visual features, preserving visual details while enhancing segmentation accuracy through linguistic guidance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.