Reproduce “Scaling Multilingual Pretraining of Pixel Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ilker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott
MPIXEL extends pixel language modeling by testing more pretraining languages, scripts, and model capacity. It processes rendered text images, avoiding a fixed character vocabulary. On SIB-200, it scores 60.1 F1 on N’ko, versus 18.2 for XLM-R and 32.1 for CANINE-S. Yet scaling does not reliably improve transfer to scripts unseen in pretraining; results describe only the tested setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.