Reproduce “Same Sentence, Different Construct: Identity-Swapped Rationales Reveal Measurement Non-Invariance in LLM Hate Speech Classifiers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenchang Wang, Yue Song
The authors test whether language models apply hate-speech labels consistently when only a targeted identity changes. They compare judgments and stated rationales on identity-controlled HateCheck templates, and test whether adding the benchmark’s definition changes the results. Hy4 has an 8.3-percentage-point excess false-negative rate for women; this finding is limited to four API endpoints on a synthetic English benchmark.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.