Reproduce “Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, Yulia Tsvetkov

This paper examines why Transformer models acquire hierarchical linguistic structures without explicit architectural biases, finding that the language modeling objective consistently enables strong hierarchical generalization. It demonstrates that Transformers implicitly favor simpler, hierarchically structured grammars to explain diverse training data, providing a Bayesian-inspired explanation for this behavior.

Key insights

  • The language modeling objective is a critical inductive bias that consistently enables strong hierarchical generalization in Transformers, resolving conflicting observations in prior research.
  • Transformers simultaneously learn and maintain distinct subnetworks within their architecture that correspond to both hierarchical and linear rule following, with training data ambiguity driving their coexistence.
  • Transformer LMs' preference for hierarchical generalization can be explained by a Bayesian 'simplicity bias,' where hierarchical grammars offer a more parsimonious (simpler and better-fitting) explanation for diverse token distributions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.