Reproduce “Volume-Preserving Transformers for Learning Time Series Data with Structure” on OpenResearch

About this paper

Benedikt Brantner, Guillaume de Romemont, Michael Kraus, Zeyuan Li

Researchers introduce the Volume-Preserving Transformer (VPT), a neural network architecture designed to learn time series data from systems governed by divergence-free vector fields, which explicitly preserves the volume of the phase space. The VPT demonstrates superior long-term stability and accuracy in predicting rigid body dynamics while conserving invariants, outperforming standard transformers that exhibit catastrophic failure.

Key insights

• Integrating geometric properties like volume preservation directly into a neural network's architecture, rather than weakly enforcing them, is critical for achieving long-term stability and physically consistent predictions in dynamical systems.
• Standard transformers fail catastrophically in long-term predictions for divergence-free systems due to a lack of structural constraints, leading to a loss of invertibility and the creation of spurious attractors.
• The Cayley transform provides an effective method for constructing volume-preserving activation functions in transformer attention mechanisms by ensuring the resulting transformation is orthogonal.

Summary by alphaXiv.