Reproduce “Quantitative Generalization Bounds for Multi-Layer Transformers: A Few Long Strings can Beat Many Short Ones” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marius Henry, Michael Rizvi-Martel, Michael Hahn
The authors analyze how training length and sample count jointly affect length generalization in fixed-precision transformers without positional encodings. Their theory shows that, under stated assumptions, increasing length from C to sC can reduce the sufficient sample count by at least s^C/2 for idealized learners. Synthetic experiments find accuracy gains from longer contexts at fixed supervised-position budgets, without identifying the cause.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.