Reproduce “What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dana Paquin, Riddhiman Jain
The authors audit whether text features predict question difficulty differently across MMLU’s STEM and non-STEM items. They estimate difficulty from responses by 1,000 models and test nine structural indicators. Entity Density is the only contrast surviving strict correction: its association is positive in STEM and indistinguishable from zero outside STEM. This item-side finding does not establish that ability scores are incomparable.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.