Reproduce “A Unified Theory of Diversity in Ensemble Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Danny Wood, Tingting Mu, Andrew Webb, Henry Reeve, Mikel Luján, Gavin Brown

Researchers at the University of Manchester and University of Bristol developed a unified theory of diversity in ensemble learning, integrating it into the statistical bias-variance decomposition. This framework precisely defines diversity as a component that subtracts from the ensemble's expected loss across various loss functions, clarifying its role in improving model performance.

Key insights

  • Diversity is intrinsically part of the bias-variance decomposition of ensemble loss, always subtracting from the expected risk for a wide range of loss functions (e.g., Bregman divergences).
  • The optimal ensemble combiner rule is specific to the loss function (e.g., arithmetic mean for squared loss, geometric mean for cross-entropy), defining a 'centroid combiner' for exact loss decomposition.
  • For non-additive losses like 0/1 loss, the 'diversity-effect' is necessarily label-dependent, formally explaining the empirical concepts of 'good' and 'bad' diversity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.