Reproduce “TIDE: Every Layer Knows the Token Beneath the Context” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho

TIDE adds shared token-identity memory to transformer layers, giving rare tokens a signal beyond their initial embedding lookup. Learned tables provide token-indexed vectors, and each layer’s router mixes them with a null option before adding the result to the residual stream. In a matched 200-billion-token comparison, rarest-decile validation loss fell 0.704 nats (9.0%), versus 0.068 nats (2.4%) for the most frequent decile; this finding is specific to that 1B-model comparison.

Key insights

  • TIDE provides K independent pathways for gradient flow to each token's embedding tables, leading to a theoretical K-fold amplification of gradient signal for rare tokens.
  • MemoryBlocks, indexed by discrete token identity, can maintain arbitrary separation between semantically distinct tokens, effectively routing around the FFN's Lipschitz continuity constraint that causes contextual collapse.
  • The architecture asymptotically generalizes the standard transformer, as the router can learn to assign sufficiently large logits to the null bank, effectively suppressing active memory contributions when not needed.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.