Reproduce “UrbanFeel: A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jun He, Yi Lin, Zilong Huang, Jiacong Yin, Junyan Ye, Yuchuan Zhou, Weijia Li, Xiang Zhang
Researchers from Sun Yat-sen University developed UrbanFeel, a comprehensive benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on their ability to understand urban development and human perception from street-view imagery. The benchmark comprises 14.3K visual questions across static scene perception, temporal change understanding, and subjective environmental perception, revealing current MLLMs significantly underperform human experts in complex temporal reasoning.
Key insights
- Current MLLMs face significant challenges with long-range temporal reasoning and integrating physical changes with perceptual shifts, performing substantially below human levels on such tasks.
- MLLMs can exhibit geographic and potential cultural biases in subjective judgments, as demonstrated by their varying responses when hypothetical city identities are introduced.
- Processing panoramic images presents a higher 'cognitive burden' for most MLLMs compared to single-view images, suggesting limitations in handling geometric distortions and complex spatial contexts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.