Reproduce “Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Optical Remote Sensing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xu Zhang, Junyao Ge, Yang Zheng, Kaitai Guo, Jimin Liang
Think2Seg-RS is a new framework designed for reasoning segmentation in optical remote sensing that decouples high-level language understanding from pixel-level mask generation by training a Large Vision-Language Model (LVLM) to generate precise geometric prompts for a frozen Segment Anything Model (SAM). Utilizing mask-only Group Relative Policy Optimization, the framework achieves state-of-the-art performance on the EarthReason dataset with 75.60% cIoU and 73.36% gIoU, and demonstrates robust zero-shot generalization.
Key insights
- Decoupling reasoning from execution effectively mitigates the 'information bottleneck' inherent in monolithic LVLM decoders, allowing for more precise pixel-level segmentation.
- Optimizing prompt generation directly with a mask-only reinforcement learning approach (GRPO) enables the LVLM to discover more effective and generalizable prompting strategies than methods reliant on noisy pseudo geometric labels.
- Smaller SAM2 variants (Tiny, Small) can outperform larger ones for semantic-level reasoning segmentation in remote sensing, suggesting that optimal segmenter scale is task-dependent and larger models may over-segment for coarser semantic annotations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.