Reproduce “Who's the (Multi-)Fairest of Them All: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration” on OpenResearch

About this paper

Karina Halevy, Karly Hou, Charumathi Badrinath

Reevaluating interpolation-based augmentation with multicalibration shows that fairness methods effective under demographic parity or equalized odds may fail for probability-sensitive fairness across many intersecting groups. On 80 Folktables state-year-task settings, ordinary Mixup generally reduced worst-group multicalibration violations more reliably than Fair Mixup, while combining Mixup with multicalibration post-processing produced the largest average gains—up to 14.22%—at some cost to balanced accuracy and additional post-processing complexity.

Key insights

• Fair Mixup does not reliably generalize from a single substantial minority group to numerous small intersections; in most multi-group settings it worsened both accuracy and multicalibration relative to the baseline.
• Interpolation itself was more consistently helpful than group balancing or explicit fairness penalties, making vanilla Mixup a stronger post-processing-free baseline.
• Multicalibration exposes fairness problems that binary group metrics can miss, particularly when models are over- or under-confident for small demographic groups.

Summary by alphaXiv.