Reproduce “YaRN: Efficient Context Window Extension of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole
YaRN extends the usable context of pretrained Llama models by selectively scaling rotary position signals and continuing training on shorter segments. A Llama 2 7B model trained on 64k-token data and extended to 128k tokens achieved sliding-window perplexity of 2.37 on ten long Proof-pile documents. This demonstrates language-modeling transfer, not general-purpose understanding at 128k.
Key insights
- Differentiating RoPE dimensions by wavelength, allowing selective interpolation ("NTK-by-parts"), effectively preserves crucial local relative order while scaling absolute positions.
- Introducing a learned pre-softmax temperature into the attention mechanism uniformly enhances perplexity across extended contexts with zero computational overhead.
- A synergistic combination of interpolation and attention scaling enables efficient transfer learning and extrapolation to context lengths longer than fine-tuning data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.