Reproduce “Bayesian Modeling of Intersectional Fairness: The Variance of Bias” on OpenResearch

About this paper

James Foulds, Rashidul Islam, Kamrun Keya, Shimei Pan

The authors develop Bayesian models for estimating fairness metrics when intersectional groups are sparse. Hierarchical logistic regression shares information across groups while representing uncertainty, then uses posterior-predictive outcome probabilities to calculate disparities. In a 1% Adult-data comparison, HLR-FB’s L1 deviation was 0.341 versus 0.572 for Bayesian logistic regression, but it was not always best across datasets and metrics.

Key insights

• Fairness auditing is itself a statistical inference problem: ratio-based metrics such as differential fairness can amplify small-sample noise, so posterior uncertainty should accompany the reported estimate.
• Posterior predictive fairness estimates can differ substantially from the posterior average of the fairness metric because fairness measures are nonlinear, especially when they involve probability ratios.
• Partial pooling and model averaging are useful safeguards in the small-data regime, but fairness conclusions remain dependent on attribute definitions, priors, preprocessing, metric choice, and model assumptions.

Summary by alphaXiv.