Reproduce “Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Liu, Hongming Li, Melissa Xiaohui Qin, Qiankun Liu, Chao Huang
Researchers introduce SEMANTICQA, a new benchmark to evaluate language models' fine-grained semantic reasoning capabilities for multiword expressions. It evaluates LMs on diverse multiword expressions across interpretation, extraction, and classification tasks, demonstrating that current models exhibit task-dependent performance and limitations in semantic consistency and workflow robustness.
Key insights
- LMs exhibit highly task-dependent semantic capabilities, with no single model achieving uniformly high performance across all operations and phrase types, indicating operation-specific strengths rather than generalized phrase-level competence.
- Evaluation metrics significantly influence the perception of model strengths; flexible, similarity-based metrics for interpretation can mask weaknesses in structurally grounded understanding measured by strict extraction metrics.
- LMs show limited workflow robustness in sequential tasks, where upstream errors (e.g., in extraction) propagate and are not reliably compensated by downstream operations (e.g., interpretation), highlighting challenges in integrating intermediate semantic outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.