Reproduce “Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Borun D Chowdhury

Research establishes that the "Lost in the Middle" U-shaped performance curve in causal decoder-only transformers is an inherent geometric property of the architecture at initialization. It provides a mathematical proof that causal masking drives primacy, residual connections create recency, and their combined effect leads to a "factorial dead zone" in the middle of the context, independent of positional encodings or training.

Key insights

  • The U-shaped positional bias is a fundamental "geometric birthright" of the causal decoder-only transformer architecture with residual connections, present at initialization before any training or positional encoding effects.
  • Causal masking rigorously guarantees a logarithmic divergence of gradient influence at the beginning of the context (Primacy Tail), while residual connections establish an isolated Dirac delta spike at the final token (Recency Anchor).
  • Between these extremes, intermediate tokens fall into a "factorial dead zone" where gradient paths suffer from fractional dilution due to the interplay of causal mixing and residual connections.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.