Reproduce “Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures” on OpenResearch

About this paper

Risham Sidhu, Julia Hockenmaier

Researchers from the University of Illinois at Urbana-Champaign developed GSU, a text-only grid dataset, to evaluate LLMs' inherent spatial reasoning across navigation, object localization, and structure composition tasks. The study revealed models struggle with egocentric frames of reference and abstracting 3D shapes, but fine-tuning smaller LLMs can achieve performance comparable to or exceeding large frontier models in these areas.

Key insights

• LLMs consistently struggle with egocentric frames of reference, often reverting to fixed cardinal reasoning and failing to update spatial perspectives over multiple steps.
• Abstracting coordinate lists into descriptions of 3D shapes proved challenging for models, which often provided vague or flattened interpretations.
• Exposure to a visual modality in Vision-Language Models (VLMs) did not consistently improve text-only spatial reasoning, especially in smaller models, suggesting visual knowledge may not generalize directly to textual spatial tasks.
• Fine-tuning smaller LLMs with synthetic reasoning traces significantly enhances their spatial reasoning, enabling them to match or surpass the performance of massive frontier models on specific GSU tasks.

Summary by alphaXiv.