Reproduce “A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning” on OpenResearch

About this paper

Soham Chatterjee, Rwitobroto Dey, Smarajit Bose

The Heterogeneous Mixture of Experts (Hetero-MoE) framework integrates diverse interpretable expert models like decision trees, linear SVMs, and QDA within a probabilistic structure. This approach enables adaptive, region-specific inductive bias selection, achieving competitive predictive performance while offering insights into local data structures through expert specialization patterns.

Key insights

• The framework provides adaptive inductive bias selection, where the model learns which interpretable model type (tree, linear, quadratic) is best suited for different local regions of the feature space.
• Expert utilization patterns act as a form of "structural diagnosis," revealing the intrinsic statistical characteristics and appropriate decision boundary types for different data regions, enhancing interpretability.
• Theoretical analysis confirms that the regression-based gating update ensures a monotone ascent for the gating auxiliary objective within the GEM algorithm, guaranteeing optimization stability.

Summary by alphaXiv.