Reproduce “Efficient Autoregressive Inference for Transformer Probabilistic Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Conor Hassan, Nasrulloh Loka, Cen-You Li, Daolang Huang, Paul E. Chang, Yang Yang, Francesco Silvestrin, Samuel Kaski and 1 more
Researchers from Aalto University and the University of Helsinki introduce a causal autoregressive buffer for set-based transformer probabilistic models, enabling efficient generation of coherent joint distributions. This method achieves 3 to 20 times faster autoregressive sampling and up to 7 times lower VRAM usage at large context sizes, matching predictive accuracy of computationally intensive baselines.
Key insights
- Decoupling static context encoding from dynamic prediction generation dramatically reduces computational complexity in autoregressive inference.
- Carefully designed attention masks and token roles enable efficient, causal autoregressive prediction while preserving permutation invariance for the initial context.
- A unified training approach, using curriculum-based masking, allows a single model to support both efficient marginal and accelerated autoregressive operations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.