Reproduce “Learning from Crowds by Modeling Common Confusions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhendong Chu, Jing Ma, Hongning Wang

Researchers at the University of Virginia introduced CoNAL, an end-to-end deep learning model that decomposes crowdsourced annotation noise into common (shared) and individual (annotator-specific) components. This model dynamically differentiates the source of confusion for each annotation, leading to consistently superior accuracy compared to existing baselines on both synthetic and real-world datasets by robustly handling pervasive common confusions.

Key insights

  • Annotation noise is not solely individual; common confusions are a prevalent and distinct source of error in crowdsourced data that degrade traditional learning models.
  • Differentiating and explicitly modeling common versus individual noise dynamically for each annotation can theoretically and empirically lead to more accurate inference of true labels.
  • An `L2`-norm regularization term is crucial for ensuring the identifiability and proper disentanglement of the common and individual noise adaptation layers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.