Reproduce “Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models” on OpenResearch

About this paper

Yuhui Zhang, Michihiro Yasunaga, Zhengping Zhou, Jeff Z. HaoChen, James Zou, Percy Liang, Serena Yeung

Researchers from Stanford University empirically investigated how negation impacts the scaling trends of large language models, observing that models exhibit inverse, U-shaped, or positive scaling on negation-heavy tasks depending on prompting methods and model family strength. The study introduced the NeQA dataset and explained these complex behaviors by decomposing the task into two sub-components: general question answering (which scales positively) and negation understanding (an emergent capability that scales sigmoidally).

Key insights

• The scaling behavior of LLMs on negation-based tasks is not static, transitioning from inverse to U-shaped to positive scaling as prompting methods become stronger or model families more capable.
• Negation understanding is an emergent capability, exhibiting a sigmoid-shaped scaling curve with a discernible transition point where models begin to reliably process negation.
• The overall NeQA scaling trend arises from the combination of linear positive scaling for non-negated question answering and the sigmoid-shaped emergence of negation understanding.

Summary by alphaXiv.