Reproduce “Extracting Interpretable Models from Tree Ensembles: Computational and Statistical Perspectives” on OpenResearch

About this paper

Brian Liu, Rahul Mazumder, Peter Radchenko

This research from MIT and the University of Sydney introduces an optimization-based framework to extract interpretable decision rules from tree ensembles, allowing joint control over the number of rules and their interaction depth. The method achieves up to a 50% increase in out-of-sample R² over existing rule extraction techniques and compresses models by up to 600x with minimal accuracy loss.

Key insights

• The ability to jointly control both the number of rules and their interaction depth is crucial for extracting interpretable models that retain high predictive accuracy from complex tree ensembles.
• Specialized optimization algorithms, including convex integer programming reformulations and iterative block coordinate descent with computational enhancements, are essential for practical application to large-scale ensembles.
• Non-asymptotic prediction error bounds demonstrate that the estimator's large-sample predictive performance is comparable to an oracle, with depth and feature weighting theoretically improving over rule weighting for deep trees.

Summary by alphaXiv.