Reproduce “Attention Projection Mixing with Exogenous Anchors” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jonathan Su

ExoFormer introduces an architectural modification to Transformer models by decoupling a dedicated "exogenous anchor" from the sequential layer stack and employing normalized mixing of query, key, value, and gate projections. This approach resolves the "first-layer tension" found in internal-anchor designs, leading to approximately 1.5 points higher downstream accuracy and requiring about 1.5 times fewer training tokens for comparable performance.

Key insights

  • Applying RMSNorm to anchor sources is critical for stable cross-layer mixing, resolving distributional mismatches that otherwise lead to instability and reduced performance.
  • The "first-layer tension" in internal-anchor designs is resolved by ExoFormer's exogenous anchors, enabling specialized roles for both the stable anchor and the sequential computational layers.
  • The "Offloading Hypothesis" posits that external anchors allow sequential layers to offload token identity preservation, thereby specializing in feature transformation and prolonging the refinement stage of information processing.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.