Reproduce “Exact Scaling Laws for a Random-Features Bigram Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Julian Enoch, Elvis Dohmatob
The authors study how representation width affects next-token prediction in a bigram model with Zipf-distributed frequencies. They freeze random embeddings and learn through a shared width-N representation, reducing parameters from d² to Nd while fitting to convergence. On WikiText-2, normalized empirical squared-loss regret reaches zero at linear width (β=1, γ=1); sublinear width retains regret. This toy-model result is not an LLM scaling law.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.