Reproduce “Training and Inference on Any-Order Autoregressive Models the Right Way” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andy Shih, Dorsa Sadigh, Stefano Ermon
MAC (Mask-tuned Arbitrary Conditional Models) refines Any-Order Autoregressive Models (AO-ARMs) by introducing a deterministic decomposition protocol and aligning the training objective with the inference-time mask distribution. This approach addresses model redundancy and objective mismatch, leading to improved likelihoods across text, image, and tabular datasets, and enhancing performance in robot shared autonomy tasks.
Key insights
- It is not necessary for AO-ARMs to learn all possible univariate conditionals from N! orderings; a minimal, deterministically selected set of conditionals is sufficient for efficient arbitrary marginal inference.
- Aligning the training distribution of univariate conditionals with their expected frequency during inference significantly improves the model's performance on downstream arbitrary conditional tasks.
- Reweighting training masks based on their cardinality can counteract the natural bias towards low-cardinality masks in induced edge distributions, leading to better generalization across different conditional contexts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.