Reproduce “Tabular foundation models for non-tabular tasks” on OpenResearch

About this paper

Goran Nakerst, John Brennan, Wouter Beugeling, Masudul Haque

Nakerst et al. from TU Dresden and collaborating institutions demonstrate that TabPFN v3, a tabular foundation model, can generalize to non-tabular classification tasks like image recognition and language identification by reformatting data into a tabular structure. The model achieved 98.6% accuracy on MNIST, 88.2% on French vs. German word differentiation, and performed above chance on Tiny ImageNet, leveraging in-context learning without task-specific training.

Key insights

• Many machine learning datasets, regardless of their native structure, can be conceptually represented and processed effectively in a tabular format.
• Pre-training TFMs on abstract causal structures may instill an understanding of relationships that allows them to generalize across diverse data types, even without explicit structural priors (e.g., spatial adjacency for images).
• In-context learning with TFMs offers efficiency and competitive performance in few-shot scenarios, reducing the need for extensive retraining and hyperparameter tuning for new tasks.

Summary by alphaXiv.