Reproduce “The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shailja Thakur, Sungeun An, Chad DeLuca, Hima Patel
IBM Research quantifies the "wording effect," revealing that meaning-preserving rephrasing causes an average 74.7 percentage point fluctuation between worst-case and best-case LLM accuracy on benchmarks. The BenchDrift framework measures both hidden capabilities (positive drift) and hidden fragilities (negative drift), showing that stronger models exhibit more negative drift and that specific linguistic and structural changes consistently impact performance.
Key insights
- The range of accuracy for an LLM on a given benchmark can vary by an average of 74.7 percentage points between its worst-case (correct on all variations) and best-case (correct on at least one variation) performance.
- Stronger LLMs generally exhibit more negative drift (loss of correct answers), indicating hidden fragilities, whereas weaker models often show more positive drift (unveiling hidden capabilities).
- Performance changes are not random; specific pragmatic and structural transformations, such as "interrogative expansion" and "persona framing," consistently induce the highest rates of negative drift across models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.