Reproduce “Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Aditya Tanna, Yash Desai, Pratinav Seth, Mohamed Bouadi, Nassim Bouarour, Vinay Kumar Sankarapu

An analysis of ensembling distinct tabular foundation models reveals a high degree of predictive redundancy among In-Context Learning (ICL) trained models, leading to a "diversity ceiling" that constrains ensemble performance gains. While one complex ensemble strategy achieved a slight accuracy improvement, it incurred a substantial increase in computational cost, suggesting a trade-off for practical deployment.

Key insights

  • Tabular Foundation Models trained with In-Context Learning (ICL) on synthetic priors form a "near-redundant pool" (mean pairwise Q-statistic of 0.961), significantly limiting the potential for performance gains from convex ensemble combiners.
  • There is a strong accuracy/compute trade-off; complex ensemble strategies like cascade stacking offer only marginal accuracy gains (+0.18%) over the best single TFM but incur drastically higher computational costs (253x).
  • Ensemble methods can degrade probability calibration, with strategies that enhance discriminative accuracy (e.g., stacking with logistic regression) often leading to poorer calibrated outputs, impacting the reliability of predictions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.