Reproduce “Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, Martin Vechev
Researchers from ETH Zurich and INSAIT conducted the first evaluation of large language models on generating rigorous natural language proofs for problems from the 2025 USA Mathematical Olympiad. The study found that most state-of-the-art models scored below 5% of the maximum points, with the highest-performing model, GEMINI-2.5-PRO, achieving only 24.4%, demonstrating fundamental shortcomings in advanced mathematical reasoning capabilities.
Key insights
- Current LLMs demonstrate fundamental limitations in logical consistency, making unjustified assumptions, and exhibiting a lack of creative problem-solving for complex mathematical proofs.
- LLMs consistently over-confidently self-assess their solutions as correct, despite significant flaws, and some models hallucinate non-existent academic references to bolster claims.
- Optimization strategies like 'answer boxing' can inadvertently lead to behavioral artifacts where models prioritize formatting over correct mathematical deduction and comprehensive proof structure.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.