Reproduce “A Controlled Study of Attention-Only Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Henry Ndubuaku, Karen Mosoyan, Jakub Mroz, Noah Cylich, Satyajit Kumar, Parkirat Sandhu, Roman Shemet, Justin H Lee

Researchers at Cactus Compute conducted a controlled study on "Simple Attention Networks" (SANs), which entirely lack Feed-Forward Networks (FFNs), revealing that when the FFN's parameter budget is reallocated to additional attention layers, the performance gap on reasoning-dense data shrinks to a minimal 0.0055 nats. The study localized the remaining performance deficit specifically to low-context query predictions, indicating FFNs primarily contribute parameter capacity for targeted knowledge recall rather than a distinct computational primitive.

Key insights

  • FFNs primarily provide parameter capacity to a "write path" into the residual stream, and their functional contribution can largely be transferred to additional attention layers when capacity is matched.
  • The FFN's performance advantage is not diffuse but highly localized to low-context query predictions, where it aids in retrieving parametric knowledge when contextual cues are insufficient.
  • QK-normalization is a critical and mandatory architectural component for ensuring the trainability and stability of deep attention-only stacks, with its removal leading to immediate divergence.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.