Reproduce “CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder” on OpenResearch

About this paper

Yongmin Lee, Hye Won Chung

CovMatch presents a multimodal dataset distillation framework that enables scalable joint optimization of both image and text encoders by reformulating the distillation objective into a closed-form solution. This approach achieves state-of-the-art performance, demonstrating up to 6.8% absolute improvement in image-text retrieval on Flickr30K and 6.1% on COCO datasets, while also providing robust cross-architecture generalization.

Key insights

• Jointly optimizing both image and text encoders during distillation is critical for achieving robust cross-modal semantic alignment and preventing performance bottlenecks in multimodal tasks.
• The computational burden of bi-level optimization in dataset distillation can be significantly reduced by approximating the inner loop with a closed-form solution, particularly by matching cross-covariance matrices.
• Combining cross-covariance matching with feature-level regularization is effective for distilling multimodal datasets, ensuring both robust relational and distributional alignment across modalities.

Summary by alphaXiv.