Reproduce “We're Afraid Language Models Aren't Modeling Ambiguity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith and 1 more
The authors introduce AmbiEnt, an English benchmark linking alternative readings to natural language inference (NLI) judgments. It contains 1,645 examples; 35.2% have multiple labels, each paired with a disambiguating rewrite. Human agreement rose from κ = 0.12 on ambiguous examples to 0.67 after rewriting. GPT-4 achieved 32.0% correctness generating readings on 50 sampled cases, so evidence is limited to the evaluations.
Key insights
- A significant portion of human annotator disagreement in NLI tasks stems directly from inherent linguistic ambiguity, which can be reliably identified by humans.
- Despite their advanced capabilities, current large language models (LMs), including state-of-the-art models like GPT-4, demonstrate substantial limitations in generating, recognizing, and consistently modeling ambiguous interpretations.
- The ability of LMs to handle ambiguity is highly dependent on the specific task design and evaluation method, indicating a lack of unified, robust understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.