Reproduce “Multilinguality in Hybrid Attention LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lucas Bandarkar, Junlin Hu, Chenyuan Yang, Mohsen Fayyaz, Nanyun Peng
The authors test whether layer order affects multilingual learning in hybrid models. Students keep a fixed number of full-attention layers, replace others with Gated DeltaNet, and learn from frozen full-attention teachers. Four alternatives beat standard periodic on held-out multilingual loss; reverse periodic reaches its 990-million-token reference gap at 345 million tokens. This evidence is limited to small distillation experiments.
Key insights
- Hybrid models offer practical advantages at long context lengths, but do not consistently outperform full-attention models on retrieval accuracy; in some evaluations, the accuracy gap narrows as context grows.
- Representation alignment in hybrid models changes sharply around full-attention layers, particularly the first one, rather than following the smoother layer-wise pattern observed in conventional full-attention models.
- All tested alternative layer layouts begin with full attention and outperform standard periodic ordering, though the experiments do not isolate first-layer placement as the cause.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.