Reproduce “Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alejandro Mozo Quesada, Sergio Gálvez Rojas, Francisco López Valverde
The authors introduce an audit for structured decision models that tests whether answers change when choices are reordered, repeats decisions, and examines probability distributions as well as selected answers. They find stereotype-aligned results alongside substantial order sensitivity; a near-neutral CrowS-Pairs average can mask extreme position effects. The audit reveals vulnerabilities but does not remove bias.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.