Reproduce “Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy” on OpenResearch

About this paper

Hosein Hasani, Mohammadali Banayeeanzade, Ali Nafisi, Sadegh Mohammadian, Fatemeh Askari, Mobin Bagherian, Amirmohammad Izadi, Mahdieh Soleymani Baghshah

The authors propose counting items by splitting a list into partitions, generating a count for each, and adding those partial counts, without fine-tuning or changing the model. In Gemini-2.5-Pro experiments on 91–100-item lists, this structured method with intermediate steps reached 0.91 exact-match accuracy, versus 0.20 to 0.79 for alternatives; results remain bounded by reliable partition sizes and text lists.

Key insights

• LLMs' default counting relies on a capacity-limited, "System-1–like" mechanism that degrades rapidly with increasing item numbers.
• The System-2 strategy works by localizing count encoding to specific final tokens within each partition, which then transfer this information via dedicated attention pathways to intermediate reasoning tokens.
• The final aggregated count is causally mediated by the numerical information stored in these intermediate representations, demonstrating a staged computational mechanism within the LLM.

Summary by alphaXiv.