Reproduce “GatedV: Self-Gated Values for Transformer Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Li-Ming Yu, Hongyu Zhou
GatedV applies SiLU to each projected attention value before weighted aggregation, leaving query-key weighting and learned parameter count unchanged. In paired FineWeb-Edu next-token tests with 256-token contexts, it lowers held-out perplexity by 2.40% in a 26.16M-parameter model, improving all three paired runs, and by 2.03% in a 2.33M model, improving all five. Larger models and downstream tasks remain untested.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.