Reproduce “Improving Position Encoding of Transformers for Multivariate Time Series Classification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Navid Mohammadi Foumani, Chang Wei Tan, Geoffrey I. Webb, Mahsa Salehi
Foumani et al. introduce ConvTran, a Transformer-based model for multivariate time series classification, which integrates two new position encoding methods: Time Absolute Position Encoding (tAPE) and Efficient Relative Position Encoding (eRPE). The model demonstrates improved performance across 32 multivariate time series datasets by better capturing temporal relationships and increasing computational efficiency compared to prior approaches.
Key insights
- Traditional position encoding methods in Transformers, such as sinusoidal absolute position encoding, exhibit issues like poor distance awareness and the 'anisotropic phenomenon' when applied to multivariate time series data.
- Adapting position encoding to specific time series characteristics, like series length and embedding dimension, improves temporal relationship capture.
- Efficient relative position encoding can be achieved by optimizing indexing strategies, significantly reducing computational overhead while retaining expressiveness for temporal relationships.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.