Reproduce “MARS: Enabling Autoregressive Models Multi-Token Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziqi Jin, Lei Wang, Ziwei Luo, Aixin Sun
The MARS method enables existing instruction-tuned autoregressive language models to perform multi-token generation through lightweight fine-tuning, avoiding architectural modifications or separate auxiliary models. This approach preserves the model's original one-token-at-a-time quality while delivering up to 1.71x wall-clock speedup in batch inference and 1.5-1.7x algorithmic speedup in multi-token mode with minimal accuracy reduction.
Key insights
- Successfully adapting autoregressive models for multi-token generation without quality degradation requires strictly preserving AR properties like causal attention, right-shifted logits, and left-to-right generation.
- An auxiliary autoregressive SFT loss is essential during fine-tuning to prevent the erosion of the original AR model's competence, especially with larger prediction block sizes.
- A dynamic confidence threshold allows for a smooth and controllable trade-off between generation speed and output quality, enabling flexible deployment tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.