Reproduce “Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Steven Moore, Huy A. Nguyen, Tianying Chen, John Stamper
The authors evaluate a hybrid rule-based system for checking multiple-choice questions against 19 item-writing criteria. It combines direct comparisons, natural-language processing, and pretrained language models, and is compared with GPT-4 and human reviewers on 200 questions. Across 3,800 criterion labels, it agrees with human classifications 90.87% of the time, versus 78.89% for GPT-4; this measures agreement, not ground-truth flaw detection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.