Reproduce “TopoBench: Benchmarking LLMs on Hard Topological Reasoning” on OpenResearch

About this paper

Mayug Maniparambil, Nils Hoehing, Janak Kapuriya, Arjun Karuvally, Ellen Rushe, Anthony Ventresque, Noel O'Connor, Fergal Reid

Researchers introduced TopoBench, a benchmark assessing Large Language Models on complex topological reasoning, revealing that even frontier models achieve limited accuracy (best 0.24 on hard instances). Their diagnostic work identified spatial parsing and maintaining structured constraint information as key bottlenecks, demonstrating that cell-aligned input formats and external tools can improve performance by 30-40 percentage points.

Key insights

• LLM failures in topological reasoning are primarily due to difficulties in extracting and maintaining structured constraint information from spatial inputs, rather than fundamental reasoning limitations.
• Premature commitment and constraint forgetting are identified as the most significant causal bottlenecks, indicating models struggle with effective backtracking and verifying semantic rules.
• The frequency of observed errors does not correlate with their causal impact on LLM performance, emphasizing the need for causal intervention analysis.

Summary by alphaXiv.