Reproduce “CodeGraph: Enhancing Graph Reasoning of LLMs with Code” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qiaolong Cai, Zhaowei Wang, Shizhe Diao, James Kwok, Yangqiu Song
CodeGraph enhances Large Language Models' (LLMs) graph reasoning abilities by converting graph problems into executable Python code, which is then run by an external interpreter. This approach led to accuracy improvements ranging from 1.3% to 58.6% on the GraphQA benchmark compared to text-based reasoning.
Key insights
- Delegating numerical and structured computation to an external code interpreter bypasses LLMs' inherent weaknesses in arithmetic and precise reasoning.
- Leveraging LLMs' strong code generation capabilities is more effective for graph reasoning than relying solely on their text-based reasoning.
- The approach is robust to different graph encoding methods (e.g., adjacency, friendship, co-authorship) and generalizes to various graph structures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.