Reproduce “Factorised Output Layers Let Small Language Models Use a 131k Vocabulary at Equal Compute” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jakub Michałek

The paper tests a two-level output head for language models with a 131k-word vocabulary. The head routes frequent words directly and groups rarer words into clusters, reducing output scoring compute. At equal training FLOPs, the two-level head and full-width adaptive softmax tie, and both outperform the flat 131k head in the d=384 comparison. This result is bounded by tokenizer and training-domain differences in this setup.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.