Reproduce “How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ahmed M. Alaa, Boris van Breugel, Evgeny Saveliev, Mihaela van der Schaar

A comprehensive, sample-level evaluation metric consisting of "u03b1-Precision", "u03b2-Recall", and Authenticity quantifies the fidelity, diversity, and generalization capabilities of generative models. This framework accurately ranks model performance, diagnoses specific failure modes, and enables post-hoc auditing to enhance the utility and privacy of generated synthetic data, particularly in sensitive domains.

Key insights

  • The proposed metrics' maximization (P "u03b1"/"u03b1" = 1 and R "u03b2"/"u03b2" = 1) if and only if generative and real distributions are identical, addressing a fundamental flaw in standard precision and recall.
  • Sample-level computability enables post-hoc model auditing, allowing for the rejection of low-quality or unauthentic synthetic samples to curate improved datasets without costly model retraining.
  • The framework offers an interpretable visualization of the privacy-utility tradeoff for generative models, critical for informed decision-making in sensitive data applications like healthcare.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.