Reproduce “Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiyuan Zhang, Danielle C. Maddix, Junming Yin, Nick Erickson, Abdul Fatir Ansari, Boran Han, Shuai Zhang, Leman Akoglu and 6 more

Mitra trains a tabular foundation model on synthetic tables from complementary causal and tree-based priors, then predicts new rows using labeled examples as context. The authors select priors by real-table performance and cross-prior tests. Across 137 datasets, Mitra with ensembling and fine-tuning scores Elo 1136 versus Attic’s 1128, a narrow lead that does not consistently extend to large-feature regression.

Key insights

  • The design of synthetic data priors is a critical, often more impactful, driver for TFM generalization than architectural innovations alone.
  • Effective prior mixtures require components that offer strong standalone performance, generate diverse data, and introduce distinctive patterns not well-covered by other priors, such as SCMs for diversity and Extra Trees for distinctiveness.
  • Pretraining TFMs on a principled mixture of synthetic data leads to superior sample efficiency and enhanced fine-tuning capabilities compared to models using heuristically designed priors.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.