Reproduce “Uncertainty in Gradient Boosting via Ensembles” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Andrey Malinin, Liudmila Prokhorenkova, Aleksei Ustimenko

Researchers introduce probabilistic ensemble techniques to quantify predictive uncertainty in Gradient Boosting Decision Tree (GBDT) models, distinguishing between data and knowledge uncertainty. The framework, including a novel "virtual ensemble" approach, demonstrates effectiveness in detecting out-of-domain inputs and enhances probabilistic predictions across various datasets.

Key insights

  • Knowledge uncertainty, derived from GBDT ensembles, proves highly effective for out-of-domain (OOD) detection, often outperforming total uncertainty measures for this specific task.
  • The proposed virtual SGLB (vSGLB) ensemble offers a computationally efficient method to obtain useful knowledge uncertainty estimates, especially for classification tasks with categorical features, without additional training or memory overhead.
  • While ensembles provide a moderate advantage in probabilistic predictions (e.g., NLL), GBDT's inherent additive ensemble structure means point prediction accuracy improvements (e.g., RMSE, error rate) over single GBDT models are often not statistically significant.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.