Reproduce “Toppling the Hierarchy in Byte-level Language Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lukas Edman, Alexander Fraser

The authors compare byte-level language models with hierarchies that group bytes into word-like units on English character-manipulation tasks. Hierarchical designs improve speed and reduce training memory, while the all-byte model scores higher on reported CUTE averages. In 24-layer experiments, Byte scores 76.6 versus 69.5 for Interleaved-12, which uses less training memory. The results omit two CUTE subtasks and are limited to tested English settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.