Reproduce “Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohamed Bouadi, Pratinav Seth, Aditya Tanna, Vinay Kumar Sankarapu
Orion-MSP presents a transformer-based architecture that enhances in-context learning for tabular data, leveraging multi-scale sparse attention and a Perceiver-style memory. The model achieves leading zero-shot performance and improved scalability, especially on high-dimensional and imbalanced datasets.
Key insights
- Combining multi-scale processing with block-sparse attention significantly improves the efficiency and expressivity of transformer-based models for complex tabular data, enabling scalability to high-dimensional datasets.
- A Perceiver-style latent memory can enable iterative refinement and cross-component communication within in-context learning architectures without violating test-data leakage constraints.
- Pretraining with diverse synthetic structural causal models effectively teaches tabular foundation models to generalize across a wide range of real-world tabular datasets and varying characteristics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.