Reproduce “Language models are not naysayers: An analysis of language models on negation benchmarks” on OpenResearch

About this paper

Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, Trevor Cohn

A comprehensive analysis evaluates large language models' understanding of negation across seven benchmarks, revealing that larger base models often perform worse on negation tasks, contrary to general scaling trends. Instruction fine-tuning, however, significantly improves models' ability to handle negation compared to similarly sized base models.

Key insights

• Scaling up language models can lead to inverse scaling for negation understanding, where larger base models perform worse due to strong memorization of factual associations.
• Base LLMs consistently struggle with fundamental lexical semantic understanding of negation, performing at chance levels on antonym/synonym tasks.
• Complex negation reasoning, particularly in Natural Language Inference tasks like downward monotonicity, remains a significant challenge for base LLMs, often resulting in below-random performance.
• Instruction fine-tuning dramatically improves LLMs' ability to handle negation across various tasks, indicating its crucial role in aligning models with complex linguistic phenomena beyond raw scale.

Summary by alphaXiv.