Reproduce “Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yingfa Chen, Zhen Leng Thai, Zihan Zhou, Zhu Zhang, Xingyu Shen, Shuo Wang, Chaojun Xiao, Xu Han and 1 more
The HALO distillation pipeline and HypeNet hybrid architecture from Tsinghua University and OpenBMB enable the conversion of large Transformer models into efficient, long-context-capable models with just 2.3 billion training tokens. This method achieves up to 3.0x decoding and 3.4x prefilling speedups at 512K context length, alongside superior long-context recall, extending operational context to 1M tokens where Transformers fail.
Key insights
- Cross-architecture distillation can be made highly data-efficient through targeted hidden state alignment and strategic selection of attention layers, requiring only a fraction of previous distillation data.
- A Hybrid Positional Encoding (HyPE) scheme, combining RoPE for RNNs and NoPE for attention layers with dynamic logits scaling, is crucial for achieving superior length generalization.
- Simpler RNN mixers like Lightning Attention, with data-independent forget gates, exhibit better length generalization compared to more complex, data-dependent RNN variants in hybrid architectures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.