Reproduce “Theorist Toolbox: Tools for Agent Based LLM-assisted economic theory Research” on OpenResearch

About this paper

Moran Koren

This research introduces a verification-first protocol for integrating large language models (LLMs) into economic theory research, addressing the challenge of LLM 'hallucinations' in mathematical reasoning. It demonstrates three distinct verification methods on a mechanism-design problem, identifying that external and adversarial checks are crucial for identifying errors, and that output polish does not correlate with rigor.

Key insights

• Convergent discovery, where independent LLM runs derive identical non-obvious mathematical objects, provides a cheap form of replication and increases confidence in the mathematical findings.
• Adversarial verification mechanisms are load-bearing, demonstrably catching subtle errors (e.g., in budget balance or incentive neutrality claims) that a single confident LLM pass would miss.
• A key dissociation exists between an output's superficial 'polish' and its actual 'rigor'; outputs with explicit `\unproven{}` flags and transparent limitation ledgers are more trustworthy than aesthetically finished but unchecked proofs.

Summary by alphaXiv.