Reproduce “Identity on Demand: Multi-Head Gated Value Residuals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Leon Lehmann
MGVR reaches lower held-out loss than a Transformer in a paired pretraining test, using token-dependent, head-specific mixing of first-layer value heads alongside gated current-layer values. At step 300, loss was 3.5572 versus 3.7605 nats per token; it reached matched loss with 20% fewer stable-phase steps. This single-run test and unsolved two-hop task support faster learning, not general scaling or mastery.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.