Reproduce “Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Christos Nikolaos Zacharopoulos, Revekka Kyriakoglou, Chara Tsoukala, Théo Desbordes

Cross-layer probing of Greek scrambling in Llama-based models shows that syntactic word-order information is most linearly accessible in middle layers, where its coding remains relatively stable, but changes format in late layers. A classifier trained on layers 20–31 transferred to early layers with below-chance performance and a strong bias toward the canonical class, suggesting directional recoding rather than simple loss of information.

Key insights

  • Post-clause representations support above-chance word-order classification, with performance peaking in intermediate layers and declining toward chance near the output layers.
  • Layers approximately 5–19 share a relatively stable coding regime, whereas late layers show weak transfer to other stages and a coordinated coefficient-sign change around layer 22.
  • Late-to-early transfer is not random: the late-trained classifier labels about 99.3% of non-canonical and 98.0% of canonical examples as canonical, producing roughly 49.3% accuracy and indicating a directional coding mismatch.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.