Reproduce “Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Max Müller-Eberstein, Rob van der Goot, Barbara Plank, Ivan Titov

Researchers from the IT University of Copenhagen, LMU Munich, and the University of Edinburgh investigated the dynamic evolution of linguistic information within language model representations during pre-training. The study introduced a methodology combining information-theoretic probing with subspace geometry measures to reveal critical learning phases, layer specialization, and how different linguistic tasks interact at a representational level.

Key insights

  • Linguistic information for many tasks emerges rapidly within the first 1,000 to 10,000 training steps, with basic tasks like POS and DEP achieving over 90% of their final F1 performance during this critical phase.
  • Representational subspaces undergo continuous and significant shifts throughout training, with early random initializations having a strong impact on overall geometry, while layer specialization (Center of Gravity) consistently localizes information across layers regardless of initialization.
  • Linguistically related tasks initially share more representational information (higher SSA similarity) during the active learning phase (10k-100k steps), but their task-specific subspaces tend to disentangle and specialize towards the end of training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.