Reproduce “LLM4ES: Learning User Embeddings from Event Sequences via Large Language Models” on OpenResearch

About this paper

Aleksei Shestov, Omar Zoloev, Maksim Makarenko, Mikhail Orlov, Egor Fadeev, Ivan Kireev, Andrey Savchenko

The LLM4ES framework leverages large language models to create robust user embeddings from event sequences by transforming structured data into enriched textual representations, followed by autoregressive fine-tuning and embedding extraction. It consistently achieves state-of-the-art or comparable performance in various user classification tasks across financial and non-financial domains, often outperforming specialized architectures.

Key insights

• LLMs can effectively learn general-purpose user embeddings from structured event sequences if the data is appropriately prepared to mitigate distributional shift.
• Text enrichment significantly enhances LLM adaptation to low-variability event sequence data, serving as an effective data augmentation strategy by providing diverse semantic cues.
• The embeddings generated by LLM4ES capture distinct and complementary information compared to specialized event sequence models, making them valuable for ensemble approaches and improving overall predictive power.

Summary by alphaXiv.