Reproduce “When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eve Fleisig, Rediet Abebe, Dan Klein
The authors present a model that predicts annotator ratings of subjective text and identifies the group the text targets, then estimates that group’s opinion from its members’ predicted ratings. This can reveal views majority vote may hide, as illustrated by a kitchen-statement example. Target-group mean-rating mean absolute error was 0.59 versus 0.76 for baseline, a 22% improvement, on U.S. and Canadian data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.