Reproduce “MalruleLib: Large-Scale Executable Misconception Reasoning with Step Traces for Modeling Student Thinking in Mathematics” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xinghe Chen, Naiming Liu, Shashank Sonkar

Researchers developed MALRULELIB, a framework and benchmark translating 101 mathematical misconceptions into executable procedures with step-by-step traces. Evaluations revealed a 19-25 percentage point gap between large language models' ability to solve problems correctly and their capacity to predict misconception-driven answers, highlighting challenges in cross-template generalization.

Key insights

  • Large language models exhibit a substantial performance gap between solving problems correctly and modeling systematic student errors, indicating distinct underlying competencies.
  • Generalizing misconceptions across different problem templates (cross-template generalization) poses a significant challenge for AI, suggesting models rely on surface patterns rather than abstract procedural understanding.
  • Providing step-by-step reasoning traces significantly improves AI's ability to predict misconception-driven answers, highlighting the value of detailed process-level data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.