Reproduce “ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli
Researchers at the University of Zurich systematically evaluated ChatGPT's zero-shot performance for various text annotation tasks, finding it consistently outperformed crowd-workers in accuracy and intercoder agreement while being significantly more cost-efficient, and achieved higher intercoder agreement than trained human annotators.
Key insights
- ChatGPT's ability to achieve high accuracy and intercoder agreement in a zero-shot setting suggests its robustness for diverse annotation tasks without prior training examples.
- The observed positive correlation between ChatGPT's accuracy and trained annotators' agreement indicates the model performs better on tasks humans also find more consistent.
- Varying ChatGPT's temperature parameter to a lower value (0.2) substantially increased its internal intercoder agreement without negatively impacting accuracy, offering a practical optimization for consistency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.