Reproduce “Nonparametric Variational Regularisation of Pretrained Transformers” on OpenResearch

About this paper

Fabio Fehr, James Henderson

Researchers at Idiap Research Institute and EPFL reinterpreted pretrained Transformers as Nonparametric Variational Bayesian models, developing a post-training regularization method. This approach, applied without further optimization, improves out-of-domain generalization performance in summarization tasks and enables models to adapt summary length based on input document information content.

Key insights

• Pretrained Transformers can be rigorously reinterpreted as Nonparametric Variational Bayesian models, providing an information-theoretic lens for their operation.
• Standard attention functions can be conceptualized as Bayesian query denoising, where attention computes the expected posterior value given a noisy query.
• Increasing uncertainty (regularization) can create a 'smooth space of models' that, while making different predictions, maintain comparable in-domain accuracy and significantly improve out-of-domain generalization.

Summary by alphaXiv.