Reproduce “Critical Phase Transition in Large Language Models” on OpenResearch

About this paper

Kai Nakaishi, Yoshihiko Nishikawa, Koji Hukushima

Research from The University of Tokyo and Kitasato University identifies a rigorous critical phase transition in Large Language Models (LLMs) at a sampling temperature T_c ≈ 1, characterized by divergent integrated correlations and power-law decaying correlations in generated text. The study reveals that natural language corpora exhibit similar critical properties, and LLMs produce natural-sounding output when operating within this critical regime.

Key insights

• Qualitative shifts in LLM generated output with varying temperature constitute a rigorous critical phase transition at approximately T_c ≈ 1, marked by the divergence of integrated correlations and power-law correlation decay.
• Natural languages themselves exhibit statistical properties consistent with criticality, and LLMs produce texts that statistically resemble natural language most closely when operating at or near their identified critical temperature.
• The observation of "critical slowing down" near T_c implies that comprehensible LLM outputs emerge during a prolonged transient phase, suggesting that natural language generation is a dynamic process occurring close to a critical point.

Summary by alphaXiv.