Reproduce “InductionBench: LLMs Fail in the Simplest Complexity Class” on OpenResearch

About this paper

Wenyue Hua, Tyler Wong, Sun Fei, Liangming Pan, Adam Jardine, William Yang Wang

Researchers introduce InductionBench, a new benchmark to evaluate the inductive reasoning abilities of large language models (LLMs) on string-to-string transformation tasks from the subregular hierarchy. The study reveals that even state-of-the-art LLMs struggle significantly with these elementary inductive problems, particularly as task complexity, defined by context window size, increases.

Key insights

• Current LLMs demonstrate a fundamental deficiency in elementary inductive reasoning, consistently failing even in the simplest complexity classes, contrasting with their performance in deductive tasks.
• The depth of sequential dependencies (context window size 'k') is a far more critical factor in task difficulty for LLMs than the breadth of symbol variation (vocabulary size).
• LLMs struggle to robustly refine hypotheses with increased novel data, often performing worse with more distinct examples, suggesting limitations in generalization rather than an ability to leverage additional information.

Summary by alphaXiv.