Reproduce “Lean-GAP: A Dataset of Formalized Graduate Algebra Problems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Seewoo Lee, Byung-Hak Hwang, Hyojae Lim, Jihoon Hyun, Ilkyoo Choi, Yeachan Park, Jineon Baek, Hyukpyo Hong and 5 more

Researchers created LEAN-GAP, a dataset of 430 formalized graduate-level algebra problems from Dummit and Foote's "Abstract Algebra" using the Lean 4 proof assistant. This work employs a multi-stage pipeline combining automated transcription and LLM-based autoformalization, demonstrating that while autoformalization systems like Codex achieve high code elaboration rates (95.5%), human review remains crucial for ensuring semantic fidelity.

Key insights

  • While LLMs can automate initial stages of formalization and achieve high syntactic success rates, they are not yet reliable for end-to-end semantic correctness in complex mathematical domains.
  • Human expertise is critical for semantic verification, acting as the most labor-intensive bottleneck in the formalization process, often identifying subtle errors and ambiguities.
  • Iterative, agent-based LLM approaches (like Codex's compiler-grounded loop) significantly improve syntactic correctness by resolving compilation issues, but semantic fidelity remains a persistent challenge.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.