Reproduce “How Well Do Vision--Language Models Understand Cities? A Comparative Study on Spatial Reasoning from Street-View Images” on OpenResearch

About this paper

Juneyoung Ro, Namwoo Kim, Yoonjin Yoon

Researchers from the Korea Advanced Institute of Science and Technology (KAIST) investigated how well state-of-the-art Vision-Language Models (VLMs) understand fine-grained spatial relationships in urban street scenes. They found that while general VLMs initially struggle with urban spatial reasoning, fine-tuning with a synthetically generated, Chain-of-Thought (CoT) supervised dataset significantly enhances their capabilities, leading to substantial improvements in complex tasks such as depth perception and negation reasoning.

Key insights

• Synthetically generated, Chain-of-Thought (CoT) supervised data can effectively adapt general-purpose VLMs for specialized domain-specific spatial reasoning tasks, especially for complex compositional questions like negation and counterfactuals.
• Different VLM architectures exhibit varied responses to fine-tuning, with some (like BLIP-2) showing substantial gains in complex reasoning, while others (like LLaVA-1.5 and InstructBLIP) can experience performance degradation on simpler perceptual tasks due to potential catastrophic forgetting.
• There is a trade-off between improving complex reasoning abilities and maintaining performance on fundamental perceptual tasks when fine-tuning general VLMs on specialized datasets.

Summary by alphaXiv.