Reproduce “UrbanPlanBench: A Comprehensive Urban Planning Benchmark for Evaluating Large Language Models” on OpenResearch

About this paper

Yu Zheng, Longyi Liu, Yuming Lin, Jie Feng, Guozhen Zhang, Depeng Jin, Yong Li

Researchers introduce UrbanPlanBench, the first comprehensive benchmark for evaluating large language models in urban planning, alongside UrbanPlanText, a domain-specific dataset. Evaluations using professional certification standards show LLMs possess substantial urban planning knowledge but generally do not meet professional human expertise levels, with Qwen2-70B being the only model to surpass the 60% certification accuracy threshold.

Key insights

• LLMs demonstrate substantial urban planning knowledge, though most fall below professional certification levels.
• Model size correlates with higher accuracy, particularly for single-choice questions.
• Performance varies significantly across subject areas, with regulatory knowledge (S3) being the most challenging for LLMs.
• Techniques like Retrieval-Augmented Generation, Chain of Thought, and Supervised Fine-Tuning substantially improve LLM performance, especially for smaller models and regulatory tasks.

Summary by alphaXiv.