Reproduce “What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dustin Wright, Sarah Masud, Jared Moore, Srishti Yadav, Maria Antoniak, Peter Ebert Christensen, Chan Young Park, Isabelle Augenstein
The authors measure how varied factual claims are across 27 language models and 155 topics. They group claims with the same meaning and use Hill-Shannon diversity (HSD), which reflects the effective number of equally likely meaning-classes. Scores generally rise across releases, and retrieval increases diversity in the tested setup. GPT-5 averages 2,621 HSD versus search’s 3,110, a lower-bound comparison for these topics and this search baseline.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.