Reproduce “PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learning” on OpenResearch

About this paper

Weihua Hu, Yiwen Yuan, Zecheng Zhang, Akihiro Nitta, Kaidi Cao, Vid Kocijan, Jinu Sunil, Jure Leskovec and 1 more

Researchers from Kumo AI and Stanford University developed PyTorch Frame, a modular framework that facilitates deep learning on multi-modal tabular data, including text and images, and enables end-to-end learning over relational databases. The framework, built upon a novel `Tensor Frame` data structure, integrates with foundation models and PyTorch Geometric, demonstrating improved performance on text-rich and relational datasets where traditional GBDTs struggle.

Key insights

• Deep learning models continue to struggle against GBDTs on conventional numerical and categorical tabular datasets, highlighting the necessity for deep learning to differentiate itself on more complex data types.
• High-quality pre-trained external embeddings (e.g., OpenAI's `text-embedding-3-large`) can achieve predictive performance comparable to or exceeding slower end-to-end fine-tuning (`text tokenized`) for multi-modal text data.
• Deep learning frameworks, particularly when combined (e.g., PyTorch Frame with PyTorch Geometric), offer significant advantages for relational databases by enabling end-to-end differentiable learning over interconnected data, a domain where GBDTs are inherently limited.

Summary by alphaXiv.