Reproduce “SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training” on OpenResearch

About this paper

Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, Tom Goldstein

SAINT is a deep learning architecture designed for tabular data that integrates intersample attention and contrastive pre-training. This approach allows the model to learn robust representations and relationships both within and across data samples, leading to performance improvements over prior deep learning methods and gradient boosting machines on diverse tabular datasets.

Key insights

• Intersample attention, a novel mechanism, allows the model to learn relationships between different data samples in a batch, providing contextual information that complements intra-sample dependencies.
• Contrastive pre-training, adapted for tabular data using techniques like CutMix and mixup, enables SAINT to learn robust representations from unlabeled data, which is especially beneficial when labeled data is scarce.
• A hierarchical attention structure, combining both self-attention (intra-row) and intersample attention (across rows), effectively captures complex patterns inherent in tabular datasets.

Summary by alphaXiv.