Reproduce “Dataset Distillation by Influence Matching” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoru Tan, Wang Wang, Sitong Wu, Xiuzhe Wu, Yangtian Sun, Chirui Chang, Shaofeng Zhang, Xiaojuan Qi
The paper introduces Influence Matching (Inf-Match), a dataset distillation framework that directly aligns the influence of a synthetic dataset with that of the full dataset on the final converged model parameters. It features a novel, efficient, and differentiable influence estimator, achieving state-of-the-art accuracy with up to 4.7% improvement on Tiny-ImageNet and demonstrating enhanced generalization across various models and multimodal tasks.
Key insights
- Directly aligning the influence of synthetic data on the final converged model parameters (outcome alignment) is a more principled and effective approach for dataset distillation than matching intermediate processes or features.
- A computationally efficient and theoretically sound data influence estimator can be constructed for deep neural networks by approximating Hessian-gradient products through unrolled optimization dynamics.
- Incorporating learnable soft labels and a progressive checkpoint sampling strategy significantly enhances the ability of distilled datasets to generalize and perform well.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.