Reproduce “Categorising SME Bank Transactions with Machine Learning and Synthetic Data Generation” on OpenResearch

About this paper

Aluffi Pietro Alessandro, Brandi Jess, Marya Bazzi, Kate Kennedy, Matt Arderne, Daniel Rodrigues, Martin Lotz

Researchers developed a machine learning pipeline for classifying Small and Medium Enterprise (SME) bank transactions, leveraging GPT-4o for synthetic data generation to address data scarcity and improve performance. The system, developed by a collaboration including the University of Warwick and financial technology partners, achieved 73.49% classification accuracy and 90.36% precision for high-confidence predictions on held-out SME data, significantly outperforming existing methods and demonstrating a 5.4x reduction in miscalibration.

Key insights

• Large Language Models (LLMs) can effectively create semantically consistent and diverse synthetic transaction data, successfully mitigating data scarcity and class imbalance without mode collapse.
• Domain-specific fine-tuning of models like FinBERT, combined with synthetic data augmentation and focal loss, significantly outperforms zero-shot LLMs and traditional methods for complex financial text classification.
• Calibration of model output probabilities is critical for practical financial applications, leading to more reliable confidence scores and better alignment with real-world label distributions.

Summary by alphaXiv.