Reproduce “Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Education” on OpenResearch

About this paper

Rui Yang, Boming Yang, Sixun Ouyang, Tianwei She, Aosong Feng, Yuang Jiang, Freddy Lecue, Jinghui Lu and 1 more

Researchers demonstrated that large language models can perform zero-shot concept graph recovery with high efficacy, often surpassing traditional supervised baselines in F1 score. They also developed CGLLM, a pipeline integrating these recovered graphs, which consistently and significantly enhanced question-answering performance on the new TutorQA benchmark for NLP education, particularly improving prerequisite prediction F1 scores by over 20 percentage points for GPT-3.5 and GPT-4.

Key insights

• Zero-shot LLMs can effectively recover scientific concept graphs, with models like GPT4-zs outperforming traditional supervised baselines in F1 score (e.g., 0.7771 vs. 0.7471).
• Grounding LLMs with a structured concept graph via the CGLLM pipeline substantially improves question-answering performance, boosting factuality and enabling more detailed, fine-grained concept identification in educational responses.
• Standard prompting enhancements like Chain-of-Thought and direct augmentation with lengthy, unstructured external documents can sometimes degrade LLM performance in graph recovery, highlighting the need for tailored strategies for structured knowledge inference.

Summary by alphaXiv.