Reproduce “Humans and LLMs Diverge on Probabilistic Inferences” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gaurav Kamath, Sreenath Madathil, Sebastian Schuster, Marie-Catherine de Marneffe, Siva Reddy
Researchers developed the PROBCOPA dataset to investigate how large language models (LLMs) perform on open-ended probabilistic inferences compared to human judgments. Their findings indicate a consistent divergence, with LLMs typically exhibiting overconfidence and less nuanced probabilistic assessments than humans.
Key insights
- Human probabilistic judgments are typically graded and unimodal for individual inferences, demonstrating nuanced agreement on interpretation despite varying likelihood estimates.
- LLMs consistently fail to produce human-like probabilistic distributions, predominantly returning extreme likelihood scores (0 or 100) and exhibiting significantly less response variation.
- LLM 'reasoning effort' (chain length) correlates with human uncertainty but does not strongly align with human cognitive load.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.