Reproduce “The Residual Stream's Effective Depth” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Barak Gahtan, Ido Galil, Alex M. Bronstein
The authors introduce effective depth, D_eff, to measure similarity among a transformer’s accumulated hidden states across layer separations, calibrated against the structural reference F_L = 2L/(L+1) for uncorrelated updates. Across 16 decoder-only language models, 15 fall below this reference; OLMo-2-7B has D_eff 1.15 and a +40.6% gap. This measures geometry, not layer use or pruning safety.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.