Reproduce “Language-model ratings of depression reflect the rater more than the patient” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Baihan Lin
The authors test how plausible language-model and scoring choices affect depression screening of the same interviews. Across 880 configurations and 189 interviews, model choice explained 30.0% of score variance, versus 10.5% for stable participant differences. Qualified raters disagreed on 40% of participants; recalibration with 40 labels halved disagreement, but left about one in five decisions different. The criterion was self-reported PHQ-8, not diagnosis.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.