Reproduce “BLiMP: The Benchmark of Linguistic Minimal Pairs for English” on OpenResearch

About this paper

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, Samuel R. Bowman

Researchers at New York University developed BLiMP, a benchmark of 67 linguist-crafted minimal pair datasets, to systematically evaluate the intrinsic linguistic knowledge of language models in English. It reveals that state-of-the-art models perform below human levels, excelling in morphological agreement but struggling with complex syntax (e.g., Island Effects) and abstract semantic phenomena.

Key insights

• Language models acquire linguistic knowledge non-uniformly; morphological agreement phenomena show steep learning curves, while complex syntax and abstract semantics exhibit very shallow learning, suggesting fundamental limitations.
• The scale of training data is a more critical factor for an LM's linguistic acuity on BLiMP than its architectural size (number of parameters), particularly for difficult phenomena.
• LMs display distinct learning biases compared to human language processing, achieving high scores in some areas but consistently falling short in others, indicating areas where human-like competence is still lacking.

Summary by alphaXiv.