Reproduce “SHAP Distance: An Explainability-Aware Metric for Evaluating the Semantic Fidelity of Synthetic Tabular Data” on OpenResearch

About this paper

Ke Yu, Shigeru Ishikura, Yukari Usukura, Yuki Shigoku, Teruaki Hayashi

A new metric, SHAP Distance, quantifies the semantic fidelity of synthetic tabular data by measuring the divergence in global feature attribution profiles between models trained on real versus synthetic datasets. This approach reveals that preserving the decision logic is more critical for the practical utility of synthetic data than achieving perfect statistical resemblance.

Key insights

• Semantic fidelity, measured by attributional consistency, is more important for the downstream predictive utility of synthetic data than achieving perfect statistical resemblance.
• SHAP Distance can identify semantic discrepancies and subtle shifts in feature importance that traditional statistical measures (like KL divergence and PCA) often overlook.
• Synthetic data generators tend to preserve global feature hierarchies but often flatten attribution distributions and simplify fine-grained nonlinear interactions or tail behaviors present in real data.

Summary by alphaXiv.