Reproduce “Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Prediction” on OpenResearch

About this paper

Jiazhen Liu, Mingkuan Feng, Long Chen

The STAMP framework introduces an "all-mask prediction" paradigm for Multimodal Large Language Models (MLLMs), effectively addressing the core trilemma of achieving high segmentation performance, fast inference, and preserved dialogue abilities simultaneously. This new approach establishes new state-of-the-art results in referring expression segmentation, with STAMP-7B achieving an average cIoU of 80.7% on RefCOCO benchmarks and 74.8% on gRefCOCO, while maintaining rapid inference speeds.

Key insights

• Decoupling dialogue generation from mask prediction allows for token-level supervision of segmentation, which preserves the MLLM's general dialogue and reasoning capabilities.
• A hybrid attention mechanism, combining causal attention for dialogue history with bidirectional attention for mask tokens, enables rich spatial context for accurate, non-autoregressive mask generation.
• Fusing image patch features and positional embeddings into mask token embeddings provides crucial visual grounding, significantly contributing to the high segmentation performance without requiring an external mask decoder during core prediction.

Summary by alphaXiv.