Reproduce “Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyuan Allen-Zhu
The authors test how Canon, a lightweight local-mixing path between nearby tokens, affects language-model performance and recall. Across 64 runs of Transformers and linear architectures at 1.3–8B parameters, Canon improves benchmark scores; an 8B Llama reaches vanilla Llama’s MMLU accuracy with about 28% fewer pretraining tokens. Yet scores do not ensure retrieval or two-hop reasoning, especially after unrelated prompt text.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.