Reproduce “Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want” on OpenResearch

About this paper

Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang and 1 more

A framework allows Multimodal Large Language Models (MLLMs) to interpret diverse visual prompts, including points, bounding boxes, and free-form shapes, for fine-grained image comprehension while retaining general image-level understanding. The approach introduces a flexible architecture, a two-stage training strategy, and a large-scale, multi-domain instruction dataset, enabling models like VP-SPHINX to achieve improved performance in referring classification and region-level captioning.

Key insights

• The Visual Prompt Encoder (VPE) effectively processes diverse visual prompts (points, bounding boxes) into unified embeddings, allowing MLLMs to understand fine-grained visual instructions.
• A two-stage training strategy, involving initial image-visual prompt-text alignment pre-training followed by multi-task instruction finetuning, is crucial for robust visual prompting comprehension.
• Leveraging advanced large language models like GPT-4V for data construction significantly improves the quality and diversity of instruction tuning data for detailed, multi-domain visual prompting tasks.

Summary by alphaXiv.