Reproduce “Autoregressive model path dependence near Ising criticality” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Hong Teoh, Roger G. Melko
This research explores how different 1D sequence orderings influence the learning efficiency of autoregressive models, specifically RNNs and transformers, when reconstructing correlations in the 2D Ising model. The study reveals that simpler paths like zigzag and snake consistently lead to faster training convergence for transformers and improved performance for RNNs, especially near criticality, with initial anisotropic correlation learning being a key factor.
Key insights
- The choice of 1D autoregressive path significantly influences the training performance and efficiency of both RNNs and transformers when modeling non-1D physical systems, particularly near criticality.
- Transformers exhibit faster overall convergence compared to RNNs, and while their final minimized NLL loss is often path-independent, the speed of convergence varies strongly with the path.
- Early-phase anisotropic learning of spatial correlations, pronounced near the critical point for certain paths (zigzag, snake), directly contributes to initial plateaus in training loss before resolution to isotropic correlations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.