Reproduce “An Information-Theoretic Analysis of In-Context Learning” on OpenResearch

About this paper

Hong Jun Jeon, Jason D. Lee, Qi Lei, Benjamin Van Roy

Researchers at Stanford, Princeton, and NYU present a general information-theoretic framework to analyze meta-learning from sequential data, deriving an elegant decomposition of prediction error into fundamental components. This framework provides a theoretical explanation for the efficiency of in-context learning in transformers, suggesting it arises from the model identifying the most suitable task-specific parameters from a pre-learned mixture of expert models.

Key insights

• In-Context Learning (ICL) in large language models can be explained as a process where the model efficiently identifies or selects the most appropriate model from a vast, pre-trained mixture of experts, rather than learning new parameters from scratch.
• The optimal Bayesian prediction error in sequential meta-learning can be generally decomposed into irreducible error, meta-estimation error (decreasing with the number of training sequences), and intra-task error (decreasing with the length of individual sequences).
• Information-theoretic tools, including rate-distortion theory, can provide robust sample complexity bounds for deep learning models, even with continuous parameters, without relying on restrictive mixing time or stability assumptions.

Summary by alphaXiv.