Reproduce “The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans” on OpenResearch

About this paper

Bella Fascendini, Kathryn McGregor, Max D. Gupta, Thomas L. Griffiths

Researchers at Princeton University introduced the "riddle riddle" paradigm to assess if Large Language Models (LLMs) possess flexible reasoning or rely on pattern matching, revealing that LLMs tend to over-apply complex reasoning strategies to structurally similar problems that only require literal interpretation. This approach demonstrated that LLMs achieved 84.9% accuracy on genuine riddles but only 50.7% on modified "riddle riddles," while human participants exhibited more adaptable strategy selection.

Key insights

• LLMs exhibit a tendency to overgeneralize inventive reasoning strategies based on surface-level features, applying them even when problems only require a simple, literal interpretation.
• Human participants, despite a default bias towards literal reasoning, demonstrate greater flexibility in adapting their problem-solving strategies based on the actual content and requirements of the task.
• For LLMs, greater memorization of a genuine riddle correlates with higher accuracy on that riddle but *lower* accuracy on its matched "riddle riddle," suggesting reliance on retrieval rather than content-driven reasoning.

Summary by alphaXiv.