Reproduce “On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction” on OpenResearch

About this paper

Yanwen Wang, Mahdi Khodadadzadeh, Raul Zurita-Milla

Researchers at the University of Twente introduce Dissimilarity Quantification by Adversarial Validation (DAV), a method that quantifies the feature space dissimilarity between sample data and prediction locations in geospatial machine learning predictions on a 0-100% scale. Their work systematically demonstrates how this quantified dissimilarity influences the accuracy of different cross-validation methods, revealing that random k-fold CV performs better with low dissimilarity, while geospatial CV methods are more suitable for intermediate dissimilarity.

Key insights

• Dissimilarity Quantification by Adversarial Validation (DAV) provides an effective and continuous metric (0-100%) for objectively measuring feature space differences in geospatial data.
• The most accurate cross-validation method varies with dissimilarity: random k-fold CV performs best at low dissimilarity, while geospatial CVs (e.g., Spatial+ CV) are more accurate for intermediate dissimilarity.
• All examined cross-validation methods exhibit significant over-optimism and fail to provide reliable error estimates when dissimilarity is very high, suggesting fundamental evaluation limits in extreme extrapolation scenarios.

Summary by alphaXiv.