Reproduce “Wisdom of the AI Crowd (AI-CROWD) for Ground Truth Approximation in Content Analysis: A Research Protocol & Validation Using Eleven Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Luis de-Marcos, Manuel Goyanes, Adrián Domínguez-Díaz
AI-CROWD asks multiple language models to independently classify the same items, uses majority vote as a provisional label, and examines agreement and disagreement as internal diagnostics. The authors tested 11 models on four English-language benchmarks. On sampled DBpedia-14 examples, the vote scored macro-F1 0.985 against benchmark labels, near GPT-5.1’s 0.987; this does not establish truth for unlabeled data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.