Reproduce “Toppling the Hierarchy in Byte-level Language Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lukas Edman, Alexander Fraser
The authors compare byte-level language models with hierarchies that group bytes into word-like units on English character-manipulation tasks. Hierarchical designs improve speed and reduce training memory, while the all-byte model scores higher on reported CUTE averages. In 24-layer experiments, Byte scores 76.6 versus 69.5 for Interleaved-12, which uses less training memory. The results omit two CUTE subtasks and are limited to tested English settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.