Reproduce “TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zi-Jian Cheng, Zi-Yi Jia, Zhi Zhou, Yu-Feng Li, Lan-Zhe Guo
Researchers from Nanjing University introduced TabFSBench, the first benchmark for systematically evaluating machine learning models, including large language models, under feature shift conditions in tabular data. Their comprehensive evaluation revealed significant performance degradation for most models when features are removed, with the extent of degradation linearly correlated with the importance of the shifted features.
Key insights
- Models generally show impaired robustness to feature shifts, with performance degradation worsening as the degree of shift increases, and no universally best performer emerging.
- A strong linear relationship exists between the collective importance of removed features and the extent of model performance degradation.
- Models performing well in closed environments also tend to exhibit better average performance in feature-shift scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.