Reproduce “GraCoRe: Benchmarking Graph Comprehension and Complex Reasoning in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zike Yuan, Ming Liu, Hui Wang, Bing Qin
GraCoRe introduces a comprehensive benchmark with a three-tier hierarchical taxonomy to systematically assess Large Language Models' (LLMs) graph comprehension and complex reasoning abilities. The evaluation revealed that LLMs struggle significantly with graph reasoning tasks, especially for complex structures and larger graphs, while OpenAI o1 exhibited superior and balanced performance across all evaluated capabilities.
Key insights
- Graph reasoning, especially for complex graph theory problems, remains a fundamental challenge and major weakness for most current LLMs.
- LLM performance on graph reasoning tasks tends to improve when semantic information (e.g., node attributes, relationship types) is available, suggesting textual context aids processing.
- Model performance is sensitive to the textual ordering of nodes in graph representations, with sequential naming yielding better results, indicating a reliance on positional cues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.