Reproduce “Language models fail at extended rule following” on OpenResearch

About this paper

Tianxiang Dai, Jonathan Fan

Researchers at Stanford University demonstrated that large language models universally possess a finite Stable Counting Capacity (SCC), reliably failing at exact multi-step rule following beyond a certain point. This limitation, observed across 126 LLM variants through a novel counting assay, indicates that models mimic procedural execution via bounded internal states that abruptly collapse, and is not overcome by increased inference-time computation or external tool use.

Key insights

• LLMs mimic procedural rule following by mapping repeated items to a finite, syntax-sensitive trajectory of internal states, which abruptly collapse when exhausted, leading to silent failures where exact information is lost.
• The fundamental limitation in procedural state preservation is not reliably overcome by increased inference-time computation (e.g., Chain-of-Thought) or external tool use, as these strategies operate on potentially already corrupted internal states.
• Traditional AI benchmarks (e.g., GPQA, SWE-bench) exhibit weak to moderate correlation with the observed Counting Capacity, indicating their insensitivity to this critical aspect of fundamental procedural reliability.

Summary by alphaXiv.