Reproduce “CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dengzhe Hou, Lingyu Jiang, Fangzhou Lin, Kazunori D Yamada
CogBattery evaluates language-model responses across 54 open-weight models and 13 text-adapted cognitive-task paradigms, comparing condition-level results with human data from each task’s source study. In 2-back, models and humans both respond “MATCH” more to lures than controls, but model false alarms to ordinary non-targets average .328 versus .071 for humans. This difference does not establish shared mechanisms or cognitive deficits.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.