Reproduce “A First-Principles Theory of Slow Thinking and Active Perception” on OpenResearch

About this paper

Hongkang Yang, Zhi-Qin John Xu, Feiyu Xiong, Weinan E

The paper develops a mathematical framework called active lifting, which represents observations with sequences of hidden thoughts and includes predictive and response-aware explanatory samplers. In a preliminary OpenWebMath experiment with Qwen2.5-7B, explanatory sampling reached normalized held-out test loss 1.7294 after 1,000 steps, versus 1.7331 predictive and 1.7345 SFT. This tests only Stage One, not broader theory.

Key insights

• Standard Transformers, acting as bounded-depth circuits, are theoretically limited in representing certain simple distributions (e.g., some HMMs), but these limitations can be overcome by leveraging appropriately structured latent spaces via 'simple projections'.
• Optimal samplers for inference (posterior sampler Q*) and training (inquisitive sampler Q_Y) are generally non-causal, indicating that existing slow-thinking models relying on causal/predictive samplers are suboptimal in terms of sample efficiency.
• The process of 'active lifting', driven by minimizing reducible uncertainty, can lead to the spontaneous emergence of an efficient, learnable 'mental language' or 'conceptual schema' for observations, resembling minimum-length coding.

Summary by alphaXiv.