Reproduce “An RKHS Perspective on Tree Ensembles” on OpenResearch

About this paper

Mehdi Dagdoug, Clement Dombry, Jean-Jil Duchamps

This paper establishes a comprehensive theoretical framework for Random Forests and Gradient Boosting by rigorously defining Random Forest kernels within Reproducing Kernel Hilbert Spaces (RKHS). It shows that infinite Random Forest predictors are unique minimizers of a penalized empirical risk in their induced RKHS and reinterprets infinitesimal gradient boosting as a gradient flow on a Hilbert manifold, introducing practical tools like Geometric Variable Importance.

Key insights

• The effective geometry and regularization properties of Random Forests are intrinsically data-dependent, encoded by the constructed RKHS, which provides a theoretical basis for their ability to adaptively mitigate the curse of dimensionality.
• Random Forest predictors implicitly regularize functions by minimizing a penalized empirical risk with respect to the RKHS norm, revealing a novel variational principle for their learning process.
• Continuous-time Gradient Boosting (Infinitesimal Gradient Boosting) can be understood as a gradient flow on an infinite-dimensional Riemannian manifold, offering a powerful geometric and continuous-time perspective on boosting's optimization path.

Summary by alphaXiv.