Reproduce “VectorLLM: Human-like Extraction of Structured Building Contours vis Multimodal LLMs” on OpenResearch

About this paper

Tao Zhang, Shiqing Wei, Shihao Chen, Wenling Yu, Muying Luo, Shunping Ji

VECTORLLM introduces a novel approach for extracting structured building contours from remote sensing imagery by leveraging a Multi-modal Large Language Model to mimic human-like, point-by-point delineation. The model achieves superior performance, notably gaining up to 13.6 AP over previous state-of-the-art methods on challenging datasets, and demonstrates robust zero-shot generalization to unseen object types and geographical features.

Key insights

• Integrating learnable positional embeddings and a pretraining stage is crucial for MLLMs to acquire precise spatial understanding and fundamental geometric reasoning for fine-grained vector extraction.
• The choice of vision encoder (e.g., RADIO or DinoV2) significantly impacts performance, with encoders retaining more low-level visual information outperforming those focused on high-level semantic alignment (like CLIP) for fine-grained tasks.
• A multi-stage training strategy including pretraining, supervised fine-tuning, and preference optimization (DPO) systematically refines the model, leading to outputs that align more closely with desired human-like contours.

Summary by alphaXiv.