Reproduce “PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
PoSE (Positional Skip-wisE) training enables efficient context window extension for Large Language Models by simulating long inputs within a fixed, shorter training window. This method allows RoPE-based LLMs to process sequences up to 128k tokens, matching full-length fine-tuning performance on language modeling and retrieval tasks with reduced computational overhead.
Key insights
- Decoupling the computational cost of fine-tuning from the target context length is achievable by simulating long inputs within a fixed, smaller training window.
- Preserving the continuity of position indices within locally sampled chunks is crucial for retaining the LLM's pre-trained language modeling capabilities.
- Dynamically sampling chunk divisions and positional biases during training exposes the model to sufficient relative position diversity to generalize to significantly longer contexts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.