Reproduce “Extending LLMs' Context Window with 100 Samples” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yikai Zhang, Junlong Li, Pengfei Liu
A novel "entropy-aware ABF" method dramatically extends large language models' context windows, enabling robust performance on 12 LongBench tasks with a macro average of 39.1. This approach demonstrates extraordinary data efficiency, achieving competitive long-context abilities with only 100 training samples and 6 steps, while maintaining performance up to 32k tokens.
Key insights
- LLMs implicitly maintain stable attention entropy in later layers when processing contexts, and scaling attention logits aids in preserving this stability for longer inputs.
- The proposed "entropy-aware ABF" method provides a robust inductive bias, facilitating efficient learning to maintain attention concentration during supervised fine-tuning.
- Training on lengthy conversational data acts as a strong starting point for context window extension, enabling effective transfer to various downstream tasks with limited in-domain data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.