Reproduce “A Divergence-Based Method for Weighting and Averaging Model Predictions” on OpenResearch

About this paper

Olav Benjamin Vassend

A new divergence-based method from the University of Inland Norway calculates model weights by integrating an optimism-penalizing prior with a combined predictive accuracy objective, demonstrating enhanced predictive performance, particularly in small-sample scenarios, and improved weight stability compared to existing ensemble techniques.

Key insights

• The method is derived as a unique empirical approximation of an ideal objective function that maximizes expected predictive accuracy on future data under a specific boundary condition.
• PAC-Bayesian inequality bounds provide theoretical justification, showing the objective estimates optimal out-of-sample weights when overfitting is negligible, and approximates regularization by the optimism-based prior when overfitting is substantial.
• The approach consistently yields model weights with systematically lower standard deviation across various sample sizes, indicating greater stability compared to stacking and negative exponentiated weighting.

Summary by alphaXiv.