Reproduce “Transolver is a Linear Transformer: Revisiting Physics-Attention through the Lens of Linear Attention” on OpenResearch

About this paper

Wenjie Hu, Sidun Liu, Peng Qiao, Zhenglun Sun, Yong Dou

Researchers at the National University of Defense Technology reformulate Transolver's Physics-Attention as a specialized linear attention mechanism, leading to LinearNO. This neural operator achieves state-of-the-art accuracy on PDE benchmarks and industrial datasets while reducing parameters by 40.0% and FLOPs by 36.2%.

Key insights

• Transolver's Physics-Attention is a specific instance of linear attention and can be mathematically derived as a Monte Carlo approximation of the continuous integral kernel operator.
• The shared learnable layers in Physics-Attention for query and key projections limit model expressiveness and lead to less diverse and less effectively utilized 'physical slices'.
• The 'slice attention' component within Physics-Attention is often redundant and can even degrade performance, especially when the linear attention mechanism is designed with more expressive, asymmetric projections.

Summary by alphaXiv.