Reproduce “Cross-Validation Is All You Need: A Statistical Approach To Label Noise Estimation” on OpenResearch

About this paper

Jianan Chen, Vishwesh Ramanathan, Tony Xu, Anne L. Martel

Researchers from the University of Toronto and Sunnybrook Research Institute developed ReCoV and fastReCoV, algorithms that detect noisy labels by analyzing performance fluctuations across repeated cross-validation folds. These methods, including the first explicit noise detection for survival analysis, improved model concordance index on medical data and achieved superior accuracy on classification tasks compared to existing techniques.

Key insights

• Fluctuations in validation performance across different cross-validation folds are not random but encode valuable information about the distribution of noisy labels.
• The occurrence frequency of noisy samples within the "worst-performing" cross-validation folds exhibits a statistically separable distribution from clean samples, enabling robust identification.
• Explicit label noise detection is crucial for empowering human experts to review and re-evaluate ambiguous cases, improving data quality and trustworthiness in high-stakes domains like medicine.

Summary by alphaXiv.