Reproduce “As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language” on OpenResearch

About this paper

Jasmine Owers, Edwin Simpson, Martha Lewis

A study from the University of Bristol and University of Amsterdam evaluated large language models' (LLMs) ability to interpret figurative language combined with negation. The research found that LLMs perform significantly worse on negated figurative expressions compared to literal negated or intensifying figurative language, with performance heavily influenced by prompting style and ranging up to human-level accuracy for some question-answering tasks on non-negated figurative text.

Key insights

• Prompting style critically impacts LLM performance; explicit question-answering generally yields higher accuracy than more implicit methods.
• The combination of figurative language and negation presents a distinct and substantial challenge for LLMs, even if they handle each component individually.
• LLMs struggle to generalize their understanding of individual linguistic features to their complex interactions in a human-like way.
• Negation is a primary linguistic feature that strongly correlates with decreased accuracy in figurative language interpretation, more so than tense or concreteness.

Summary by alphaXiv.