Reproduce “Comparison of Bayesian predictive methods for model selection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Juho Piironen, Aki Vehtari
Aalto University researchers quantitatively compared Bayesian model selection methods for variable selection, demonstrating that the projection predictive method offers robust simplification of complex models with minimal loss in predictive accuracy. This method, when complemented by an outer cross-validation step for model size determination, consistently yields stable submodels. Bayesian Model Averaging consistently achieved the best overall predictive performance.
Key insights
- Bayesian Model Averaging (BMA) consistently offers the best predictive performance, indicating that submodel selection should prioritize simplification over predictive improvement.
- Direct optimization of generalization utility estimates (e.g., from cross-validation or information criteria) is highly susceptible to overfitting and selection-induced bias, particularly with limited data.
- The projection predictive method robustly transfers information from a comprehensive reference model, enabling the creation of parsimonious submodels with stable and near-optimal predictive performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.