Reproduce “Large language models for aspect-based sentiment analysis” on OpenResearch

About this paper

Paul F. Simmering, Paavo Huoviala

This research from Q Agentur für Forschung GmbH demonstrates that fine-tuned GPT-3.5 models achieved a new state-of-the-art F1 score of 83.8% on the SemEval-2014 Aspect-Based Sentiment Analysis benchmark, outperforming existing specialized models. The study details how prompt engineering, in-context learning, and fine-tuning impact performance and operational costs for generalist large language models on this complex, structured NLP task.

Key insights

• Fine-tuning generalist LLMs enables them to achieve state-of-the-art performance on complex structured NLP tasks, making them robust to variations in prompts and in-context examples.
• Many discrepancies between LLM outputs and benchmark gold standards are valid common-sense interpretations that diverge from specific annotation rules, indicating that benchmark performance may not always directly correlate with practical utility.
• For non-fine-tuned models, 6 in-context examples often provide optimal performance, and enforcing structured JSON output significantly improves result quality.

Summary by alphaXiv.