Reproduce “Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory” on OpenResearch

About this paper

Peng Cui, Heejin Do, Mrinmaya Sachan

This research applies Knowledge Space Theory (KST) to evaluate whether large language models (LLMs) exhibit coherent, human-like knowledge structures in mathematical reasoning. It finds that LLMs show systematically lower adherence to prerequisite knowledge dependencies compared to human learners, even with high accuracy, suggesting a fragmented internal knowledge organization and a reliance on pattern matching over hierarchical understanding.

Key insights

• LLMs demonstrate systematically lower adherence to prerequisite knowledge dependencies (lower Prerequisite Satisfaction Ratio) compared to human learners, even when achieving comparable or higher overall accuracy.
• Prerequisite-based in-context learning does not consistently offer a greater performance advantage for LLMs than other retrieval strategies, suggesting a reliance on contextual patterns rather than activating a structured, hierarchical knowledge base.
• LLMs, particularly closed-source models, do not exhibit a consistent, shared knowledge structure among themselves, and their knowledge growth patterns diverge from the cumulative, hierarchical nature observed in humans.

Summary by alphaXiv.