Reproduce “Dataset Distillation: A Comprehensive Review” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ruonan Yu, Songhua Liu, Xinchao Wang

This paper presents a comprehensive review of Dataset Distillation (DD), a field focused on compressing large datasets into smaller synthetic ones that can achieve comparable model performance. The review by researchers from the National University of Singapore unifies the field by proposing a systematic taxonomy based on optimization objectives and reveals theoretical interconnections between different DD methods, while also empirically benchmarking their performance, efficiency, and cross-architecture generalization.

Key insights

  • Dataset Distillation methods can be systematically categorized into three main optimization objectives: Performance Matching, Parameter Matching, and Distribution Matching.
  • Theoretical connections exist between these distinct objectives, for instance, Performance Matching can be equivalent to optimal parameter matching, and gradient matching relates to first-order distribution matching.
  • The use of differentiable augmentations (like DSA) and specific parameterization strategies for synthetic data are crucial for improving distillation performance and generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.