Reproduce “K2-Horizon: A Frontier Fleet of Open Source Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
K2 Team, Zhengzhong Liu, Haonan Li, Xuezhe Ma, Alexander Moreno, Subham Sekhar Sahoo, Rupesh Kumar Srivastava, Mikhail Yurochkin and 87 more
K2-Horizon releases six models with checkpoints, logs, training data, and code. The authors train four dense and sparse models on a shared 22-trillion-token corpus; their normalized loss curves nearly overlap, though normalization aligns endpoints. At final mid-training, 36B-A4B scores 86.9 on 0-shot MMLU versus 88.1 for dense 32B, with about one-seventh as many active backbone parameters per token.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.