Reproduce “FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data” on OpenResearch

About this paper

Dongyu Zhang, Liang Wang, Xin Dai, Shubham Jain, Junpeng Wang, Yujie Fan, Chin-Chia Michael Yeh, Yan Zheng and 2 more

FATA-Trans introduces a Transformer architecture for sequential tabular data that explicitly distinguishes between static and dynamic fields and incorporates both record order and time intervals. This design achieves superior performance with AUCs up to 0.9992 in anomaly detection and 0.8057 in review rating prediction, while notably reducing pre-training time compared to prior Transformer-based methods.

Key insights

• Differentiating and separately processing static and dynamic fields significantly reduces computational redundancy and enables the model to learn more robust and meaningful representations.
• Explicitly incorporating time intervals, beyond just sequence order, through a learnable time-aware position embedding is crucial for capturing nuanced behavioral patterns in sequential data.
• Pre-training with a masked language modeling objective, tailored to the field-type and time-aware input representations, is vital for achieving high performance on downstream tasks.

Summary by alphaXiv.