Reproduce “High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models” on OpenResearch

About this paper

José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan

This paper investigates the reliability of Tabular Foundation Models (TFMs) by benchmarking their uncertainty quantification against traditional methods like Gradient-Boosted Decision Trees (GBDTs). It found TFMs achieve higher predictive accuracy but exhibit lower conditional coverage, indicating a tendency for overconfident errors, particularly in challenging data regimes.

Key insights

• A fundamental trade-off exists between predictive accuracy and uncertainty reliability in tabular learning, where higher accuracy from TFMs often comes at the cost of less trustworthy uncertainty quantification.
• Tabular Foundation Models frequently exhibit 'overconfident errors,' meaning their narrow, high-confidence prediction sets often fail to include the true label more frequently than desired.
• Gradient-Boosted Decision Trees (GBDTs) consistently provide more dependable and better-calibrated uncertainty estimates, offering a more balanced trade-off between accuracy and reliability, especially under noisy or complex data conditions.

Summary by alphaXiv.