Reproduce “Enhancing Transfer Learning with Flexible Nonparametric Posterior Sampling” on OpenResearch

About this paper

Hyungi Lee, Giung Nam, Edwin Fong, Juho Lee

KAIST AI and The University of Hong Kong researchers developed Nonparametric Transfer Learning (NPTL), a flexible posterior sampling method for deep models that integrates nonparametric learning to enhance robustness against distribution shifts and model misspecification in transfer learning. NPTL consistently improved predictive accuracy and uncertainty quantification across vision and language tasks while offering a parallelizable inference process.

Key insights

• Modeling a nonparametric prior over the data distribution offers a more flexible and robust approach to Bayesian inference in transfer learning than direct parametric priors on model parameters, especially under distribution shifts.
• Pre-trained models can effectively inform a nonparametric prior by generating 'pseudo-data' from their predictions on downstream inputs, leveraging upstream knowledge while adapting to downstream specifics.
• The use of a block Dirichlet distribution is crucial for achieving numerical stability and practical feasibility when applying nonparametric posterior sampling to large-scale deep learning models and datasets.

Summary by alphaXiv.