Reproduce “Position: Understanding LLMs Requires More Than Statistical Generalization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Patrik Reizinger, Szilvia Ujváry, Anna Mészáros, Anna Kerekes, Wieland Brendel, Ferenc Huszár

The authors show that matching an autoregressive language model’s test distribution does not determine its behavior on downstream tasks. They compare models trained on a synthetic grammar, where out-of-distribution prompts test whether completions preserve equal counts of a’s and b’s. At similar validation loss, maximum-likelihood training achieves 43.7% rule accuracy; this finding is limited to the tested synthetic setup.

Key insights

  • Autoregressive probabilistic models are inherently non-identifiable, meaning functionally equivalent models (with similar test loss) can exhibit vastly different behaviors, especially on out-of-distribution inputs.
  • Functional non-identifiability highlights that limited support in training data renders model behavior on zero-probability (OOD) inputs undefined by the likelihood function.
  • Epsilon-non-identifiability shows that even an infinitesimally small difference in KL divergence can mask radical differences in properties like in-context learning, particularly for low-probability sequences.
  • Parameter non-identifiability explains how functionally identical models can possess distinct internal parametrizations, leading to varied optimization landscapes and downstream transferability/fine-tunability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.