Reproduce “A Note on "Assessing Generalization of SGD via Disagreement"” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andreas Kirsch, Yarin Gal
Andreas Kirsch and Yarin Gal of the University of Oxford provide a critical re-evaluation of the Generalization Disagreement Equality (GDE) proposed by Jiang et al. (2022), offering a simplified theoretical framework. Their empirical analysis shows that the GDE's associated calibration metrics degrade significantly when prediction disagreement is high and under distribution shifts, which undermines the practical application of GDE for estimating generalization without labels.
Key insights
- The Generalization Disagreement Equality (GDE) and its proofs can be significantly simplified, revealing that GDE holds if a model is class-aggregated calibrated.
- Calibration metrics, including those associated with GDE, consistently degrade (increase) when prediction disagreement is high and worsen under distribution shifts.
- The need for labels to estimate the required calibration metrics creates a "circular argument," undermining the practical applicability of GDE for truly label-free generalization assessment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.