Reproduce “Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction” on OpenResearch

About this paper

Mathieu Blondel, Michael E. Sander, Germain Vivier-Ardisson, Tianlin Liu, Vincent Roulet

The authors show that an autoregressive model can represent a sequence distribution defined by an energy-based model when each next-token score includes the soft value of continuations. In finite function space, they prove an exact, invertible correspondence and equal minimum negative log-likelihoods. Tiny synthetic experiments show close training curves; they do not establish practical planning or learning future values.

Key insights

• Autoregressive models, at their functional optimum, are implicitly equivalent to energy-based models, with their next-token prediction inherently capturing future sequence-level information via a soft Bellman equation.
• The "lookahead" capability of ARMs is not fundamentally limited by sequential generation; their local next-token scores effectively encode the soft value of all future continuations, resembling dynamic programming.
• Supervised learning using negative log-likelihood (teacher forcing) for ARMs is functionally equivalent to optimizing an EBM, demonstrating that ARMs can perfectly represent target EBM distributions in function space.

Summary by alphaXiv.