Reproduce “A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answering” on OpenResearch

About this paper

Larissa Pusch, Alexandre Courtiol, Tim Conrad

A human-in-the-loop, LLM-centered architecture was developed to allow natural language querying of Knowledge Graphs by generating, explaining, and interactively amending Cypher queries. The framework demonstrated that LLMs can provide accurate query explanations and, with iterative refinement, generate correct Cypher queries for both synthetic and real-world domain-specific KGs, with top models achieving 100% accuracy on some tasks.

Key insights

• LLMs can generate natural language explanations for Cypher queries and detect certain flaws, which enhances transparency and user trust in the reasoning process, although models may struggle with specific error types like temporal information omissions.
• The interactive amendment loop significantly improves the overall accuracy of LLM-generated Cypher queries, demonstrating the value of user feedback in bridging the gap between natural language intent and formal query languages.
• LLM performance in KG question answering varies considerably across different models and question complexities, with real-world, expert-formulated questions often posing greater challenges than synthetic or simpler benchmarks.

Summary by alphaXiv.