Reproduce “Negative Pre-activations Differentiate Syntax” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Linghao Kong, Angelina Ning, Micah Adler, Nir Shavit
The authors test whether negative pre-activations in MLP neurons support grammatical behavior. They select neurons whose distributions differ from a unit-Gaussian reference by Wasserstein distance, then zero negatives in the top 1% per layer. On Llama 3.1 8B, TSE accuracy falls from 84.4% to 55.2%, versus 85.9% for a perplexity-matched control; this is benchmark evidence, not proof of grammar loss in every context.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.