Reproduce “CoT4Det: A Chain-of-Thought Framework for Perception-Oriented Vision-Language Tasks” on OpenResearch

About this paper

Yu Qi, Yumeng Zhang, Chenting Gong, Xiao Tan, Weiming Zhang, Wei Zhang, Jingdong Wang

CoT4Det introduces a Chain-of-Thought framework for Large Vision-Language Models (LVLMs) to enhance performance on perception tasks by reformulating them into multi-step language-aligned reasoning. The approach achieved an 8.3 mAP improvement on COCO-Val for object detection (ground-truth categories) and a 13.6 mAP gain (full-category setting) over the Qwen2.5-VL-7B baseline, while largely preserving general vision-language capabilities.

Key insights

• Reconceptualizing complex perception tasks into interpretable classification, counting, and grounding steps allows LVLMs to leverage their intrinsic language reasoning capabilities more effectively, mitigating issues like redundant predictions and failure to reject non-existent objects.
• Chain-of-Thought (CoT) acts as a critical reasoning scaffold, enabling LVLMs to better utilize fine-grained information from high-resolution visual inputs, particularly for improving detection of small objects.
• A balanced training approach, combining diverse detection-style annotations with general vision-language data, successfully enhances perception performance without significantly degrading the model's broader multimodal understanding.

Summary by alphaXiv.