Reproduce “Bayesian leave-one-out cross-validation for large data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Måns Magnusson, Michael Riis Andersen, Johan Jonasson, Aki Vehtari
A scalable Bayesian leave-one-out cross-validation (LOO-CV) method is introduced for large datasets, combining importance sampling with approximate posteriors and probability-proportional-to-size subsampling. This approach reduces computational complexity from O(nS) to O(n + mS), enabling robust model evaluation with diagnostics while demonstrating theoretical consistency and empirical precision gains over simple random sampling.
Key insights
- The proposed estimator for elpd_loo is theoretically consistent for large datasets, even when using approximate posteriors and fixed subsample sizes.
- PPS subsampling provides significantly higher precision for LOO-CV estimates compared to simple random sampling, allowing for much smaller subsamples.
- The k-hat diagnostic from PSIS effectively indicates the reliability of importance sampling estimates, providing guidance on potential approximation failures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.