Reproduce “A Foundation Model for Multimodal Event Sequences in Financial Applications” on OpenResearch

About this paper

Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin, Gleb Zaripov, Alexander Uglov, Alexey Vasilev, Anton Klenitskiy

Sber AI Lab developed a transformer-based foundation model that processes multimodal event sequences like financial transactions, clickstream, and communications. This model learns universal user embeddings, which, when combined with existing tabular features, measurably improves predictive performance in financial applications, demonstrating a 1% uplift in Net Present Value in production A/B tests.

Key insights

• Early fusion of heterogeneous event attributes within a transformer backbone effectively learns comprehensive, universal user representations from diverse financial data.
• Self-supervised pretraining with a Next-Event Prediction objective consistently outperforms supervised end-to-end training, particularly when unlabeled data is more abundant than labeled data.
• Integrating learned representations from event sequences with established, rich engineered tabular features yields the most robust performance, showing that new approaches can complement existing domain knowledge.

Summary by alphaXiv.