Reproduce “Leveraging Annotator Disagreement for Text Classification” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jin Xu, Mariët Theune, Daniel Braun

The authors compare methods for using annotator disagreement in hate-speech and abusive-conversation classification. A probability-based model trains on label proportions, alongside annotator-style ensembles and instruction tuning. On hate speech, it has lower test cross entropy than the ensemble; on abusive conversation, instruction tuning is lowest. Survey participants preferred its distributions, but this measures preference, not accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.