Reproduce “Transfer Learning with Deep Tabular Models” on OpenResearch

About this paper

Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, Micah Goldblum

Deep tabular models with transfer learning consistently outperform traditional Gradient Boosted Decision Trees, including those enhanced with stacking, especially in low-data environments such as medical diagnosis with as few as 4-20 samples. The research demonstrates that deep tabular models can learn reusable features, with supervised pre-training proving more effective than self-supervised approaches, and introduces a pseudo-feature method to address differing feature sets between tasks.

Key insights

• Deep tabular neural networks can learn effective transferable representations, challenging the historical dominance of GBDT models in data-scarce scenarios.
• Supervised pre-training on related tasks is more effective for tabular transfer learning than current self-supervised learning methods like Masked Language Modeling or contrastive learning.
• Practical strategies, such as the proposed pseudo-feature method, can enable knowledge transfer even when upstream and downstream feature sets are misaligned.

Summary by alphaXiv.