Reproduce “Learning to Learn a Language” on OpenResearch

About this paper

Lennart Carstens-Behrens, Holger Fröhlich

PFLM predicts next-byte patterns in real text without pretraining on real-language examples. It trains on 150 billion synthetic tokens from freshly sampled recurrent structural causal models, then adapts through context with frozen weights. On six Wikipedia languages, prediction cost falls from eight bits to 0.9–2.4 bits per byte at one million bytes, but remains behind modern models trained on trillions.

Key insights

• Because each training sequence comes from a newly sampled process, successful prediction requires using the observed prefix to infer its regularities rather than memorizing a fixed training language.
• The synthetic prior reproduces selected text-like statistics, including slowly decaying dependencies; making all latent nodes update at every step reduces integrated long-range mutual information by 18%.
• Transfer is uneven: the model predicts some unfamiliar languages and data types better with more context, but computability alone is insufficient, as it performs at chance on the binary digits of π.

Summary by alphaXiv.