Reproduce “Good Classification Measures and How to Find Them” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Martijn Gösgens, Anton Zhiyanov, Alexey Tikhonov, Liudmila Prokhorenkova
This paper systematically analyzes classification performance measures using an axiomatic framework to define desirable properties. It establishes an impossibility theorem revealing trade-offs between key properties and proposes new measures like Symmetric Balanced Accuracy and Correlation Distance, which better satisfy these theoretical desiderata.
Key insights
- An impossibility theorem demonstrates that Monotonicity, Distance, and Exact Constant Baseline cannot be simultaneously satisfied by a binary classification measure, indicating inherent trade-offs.
- Many common measures like accuracy and F-measure lack a "constant baseline," making them susceptible to bias from class imbalance or predicted class sizes.
- Measures satisfying more desirable theoretical properties, such as Matthews Correlation Coefficient (MCC) and the newly proposed Symmetric Balanced Accuracy (SBA), show greater empirical consistency across different evaluation scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.