Reproduce “CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, Chris Welty
The CrowdTruth 2.0 framework introduces refined quality metrics for crowdsourcing, moving beyond traditional agreement-centric approaches to interpret inter-annotator disagreement as a valuable signal. It formally defines and iteratively calculates mutually dependent scores for worker quality, media unit ambiguity, and annotation clarity, providing a more nuanced understanding of truth in complex annotation tasks.
Key insights
- Inter-annotator disagreement in crowdsourcing tasks should be interpreted as a valuable signal, not just noise, to understand inherent data ambiguity and diverse perspectives.
- Explicitly modeling the mutual dependencies between worker quality, media unit ambiguity, and annotation quality is crucial for accurately diagnosing the sources of disagreement.
- A comprehensive quality assessment requires an iterative refinement of metrics, where the quality of one component influences and is influenced by the others.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.