Reproduce “Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Aengus Lynch, Gbètondji J-S Dovonon, Jean Kaddour, Ricardo Silva

Spawrious is a synthetic dog-image benchmark testing whether classifiers rely on class-correlated backgrounds. It tests one-to-one associations and many-to-many group associations, reversing the latter at test time. On M2M-Hard, none of six ImageNet-pretrained ResNet50 methods exceeds 70% accuracy: CausIRL reaches 68.93%, versus ERM at 58.70%. These results show difficulty on this benchmark, not that correlation structure caused the gap.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.