Reproduce “Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jingyun Jia, Chandan Singh, Rich Caruana, Ben Lengerich

TabDistill, developed by researchers at the University of Wisconsin–Madison and Microsoft Research, enhances Generalized Additive Models by systematically extracting complex feature interactions from Tabular Foundation Models using post-hoc attribution. This method improves predictive accuracy for interpretable models on tabular datasets, achieving better performance than baseline interaction selection techniques.

Key insights

  • Tabular Foundation Models (TFMs) implicitly capture complex, high-order, and context-dependent feature interactions effectively, even in low-data and noisy conditions.
  • Model-agnostic post-hoc attribution techniques, particularly FBII computed via SPEX, can reliably and efficiently distill these specific interactions from opaque TFM predictions.
  • Explicitly integrating these distilled interactions into interpretable GAMs significantly boosts their predictive accuracy while preserving their inherent transparency, offering a practical balance between performance and interpretability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.