Reproduce “Generative Criticality in Large Language Model Temperature Scaling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huajian Ruan, Jinyang Li, Xingyu Guo, Lingxiao Wang
Researchers developed a statistical-field framework to analyze large language model (LLM) text generation, discovering phase-transition-like critical behavior as a function of the softmax temperature. This behavior is evidenced by a divergent susceptibility, a sharp change in an order parameter, and a minimum in the intrinsic dimension of generated text embeddings, particularly around T_c ≈ 1.4 for Qwen3 models.
Key insights
- LLM generative processes exhibit signatures analogous to critical phenomena or phase transitions observed in physical systems, controllable by the softmax temperature.
- The softmax temperature acts as an effective "thermodynamic" parameter that controls the collective statistical structure and phase of LLM outputs.
- The intrinsic dimension of the generated text embeddings provides an independent geometric indicator that correlates with the identified critical region and phase transitions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.