Reproduce “Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI” on OpenResearch

About this paper

Nicholas Pangakis, Samuel Wolken

A human-centered framework for evaluating generative AI in text annotation demonstrates GPT-4's variable performance across computational social science tasks, achieving a median F1 score of 0.707 on non-public datasets. The evaluation revealed that while GPT-4 shows stronger recall, its performance is often surpassed by fine-tuned supervised models when adequate human-labeled training data is provided.

Key insights

• Generative LLM performance in text annotation is highly variable across tasks, even within the same dataset, necessitating task-specific human validation rather than assuming generalized capabilities.
• LLMs tend to exhibit stronger recall than precision, suggesting they are more effective at identifying most positive instances but are prone to generating false positives.
• Consistency scores, derived from repeated LLM annotations, can effectively identify ambiguous or 'edge-case' classifications that warrant human review, thereby improving overall annotation quality.

Summary by alphaXiv.