Reproduce “ImputeFormer: Low Rankness-Induced Transformers for Generalizable Spatiotemporal Imputation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tong Nie, Guoyang Qin, Wei Ma, Yuewen Mei, Jian Sun

ImputeFormer, developed by researchers from Tongji University and The Hong Kong Polytechnic University, is a Transformer-based model that integrates low-rank factorization and a Fourier Imputation Loss for generalizable spatiotemporal data imputation. It achieved the lowest Mean Absolute Error across various traffic benchmarks, improving point missing by 4.4% to 20.2% and block missing by 7.9% to 26.5% compared to leading baselines, while training approximately 15 times faster than other Transformer models.

Key insights

  • Integrating strong inductive biases like low-rankness from traditional models with the high expressivity and learning capabilities of Transformers can overcome limitations of both isolated approaches in spatiotemporal imputation.
  • Explicitly guiding Transformer learning with structural priors through projected attention and a Fourier sparsity loss mitigates issues such as noise preservation and high-rank estimations, leading to enhanced generalizability and robustness.
  • Novel low-dimensional projections and embedded attention mechanisms enable efficient, low-rank factorization of attention in Transformer architectures, effectively preventing spurious correlations and improving scalability for sparse spatiotemporal data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.