Reproduce “Filtering Beats Fine Tuning: A Bayesian Kalman View of In Context Learning in LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Andrew Kiruluta

A "theory-first" framework from UC Berkeley models in-context learning in LLMs as online Bayesian state estimation via Kalman filtering, demonstrating that epistemic uncertainty rapidly contracts with informative prompt tokens, which provides a principled explanation for few-shot generalization and positions optimization as a degenerate limit.

Key insights

  • Epistemic uncertainty, represented by posterior covariance, contracts exponentially fast as informative prompt tokens accumulate, providing a principled explanation for few-shot generalization.
  • Classical gradient descent and natural gradient methods are revealed as singular, noise-free limits of the more general Bayesian filtering dynamics.
  • Prompt informativeness can be theoretically linked to observability conditions and information accumulation, offering a foundation for systematic prompt design.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.