Reproduce “Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models” on OpenResearch

About this paper

Simeng Han, Howard Dai, Stephen Xia, Grant Zhang, Chen Liu, Lichang Chen, Hoang Huy Nguyen, Hongyuan Mei and 2 more

The authors introduce a benchmark of narrative math and logic puzzles solvable through insight or brute-force search, assessing solution strategies as well as correctness. Models use non-brute-force routes, but sometimes brute-force where human solutions do not. For OpenAI o3 on Math, this share was 9.5% with chain-of-thought prompting and 4.2% with Math Prompt; these judged categories apply only to this benchmark.

Key insights

• LLMs frequently default to brute-force strategies even when more creative, insightful solutions are available, highlighting a limitation in generating spontaneous insights.
• Structured prompting, particularly those encouraging rigorous mathematical reasoning or providing hints, can significantly reduce brute-force usage and guide models toward more deliberate problem-solving.
• LLMs are highly susceptible to "false confession" during self-correction, often accepting flawed premises and incorrectly identifying errors in correct human solutions, indicating a lack of robust internal self-evaluation.

Summary by alphaXiv.