Reproduce “Universal priors: solving empirical Bayes via Bayesian inference and pretraining” on OpenResearch

About this paper

Nick Cannella, Anzo Teh, Yanjun Han, Yury Polyanskiy

This work theoretically justifies the observed effectiveness of pretrained transformers in Empirical Bayes (EB) problems, demonstrating that specific "universal priors" in a hierarchical Bayesian framework enable robust adaptation to diverse test distributions. The research further explains transformer length generalization as implicit "alpha-posterior" inference, where regret bounds saturate at a rate dependent on training length rather than test length.

Key insights

• The effectiveness of pretrained transformers in EB can be statistically explained by their implicit approximation of hierarchical Bayes estimators trained under 'universal priors' that induce strong posterior contraction around the true underlying prior.
• Hierarchical Bayes estimators are shown to be admissible and can achieve minimax regret for EB problems, providing a strong theoretical foundation for the PoP-based pretraining approach.
• Transformer length generalization is linked to 'alpha-posterior' inference, where the model adapts to varying sequence lengths by effectively performing Bayesian inference with a fractional observation strength proportional to the training-to-test length ratio.

Summary by alphaXiv.