Reproduce “Type Diversity Enables Transformers to Generalise Compositionally” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anssi Moisio, Mathias Creutz, Mikko Kurimo
Type diversity—the number of distinct grammatical constructors available for a linguistic category—strongly influences how Transformers generalise to unseen compositions. Experiments on COGS, SLOG, and SCAN show that expanding diversity within a target type can raise structural-generalisation accuracy from near-zero to above 90%, bringing lexical and structural generalisation onto broadly similar performance curves.
Key insights
- Increasing diversity among constructors of the directly tested type consistently improves generalisation to a held-out constructor; in COGS, artificial NP diversification raised accuracy to 92.4% from roughly 2%.
- SCAN experiments show similar diversity–accuracy trends for lexical verbs, verb phrases, and higher-level imperative structures, challenging the view that structural generalisation is inherently more difficult for Transformers.
- Compound divergence is not intrinsically a difficulty measure: it can increase alongside accuracy when caused by richer training support, rather than by increasingly unfamiliar test combinations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.