Reproduce “Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pedro Carvalho Brom
The tutorial presents a pipeline for building and evaluating a small language model from scratch on specialist text using one consumer GPU. It trains the same Transformer autoregressively and with masked diffusion. Across two Brazilian Portuguese corpora, autoregressive validation performance worsens after 64 epochs while diffusion improves through 256; this pattern is limited to the tested model, corpora, and budgets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.