Reproduce “Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinwen Luo, Han Jiang, Ziang Xiao, Susu Zhang
The authors use LSIRM to map language models against benchmark items and estimate how correct-answer categories contribute to performance scores. In BBH, removing that fitted contribution while holding estimates fixed moves Phi from rank 396 to 814 and DeepSeek from 344 to 97 among 1,348 models. The result shows rankings depend on score components, but not which ranking measures ability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.