Reproduce “Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Models” on OpenResearch

About this paper

Junjie Li, Ziao Wang, Jianghong Ma, Xiaofeng Zhang

Researchers developed the Capability-Attributed Data Curation (CADC) framework for Vision-Language Models, which curates data based on the model's intrinsic learning capabilities rather than external task definitions. This method achieved performance surpassing full-data training using only 5% of the data, demonstrating enhanced efficiency and controllability in instruction tuning.

Key insights

• Intrinsic capabilities, discovered from learning dynamics, offer a more interpretable and effective decomposition of a VLM's skills than external, human-defined task labels, as external definitions can misalign with how models truly learn.
• A single training sample can influence multiple intrinsic capabilities, and instruction datasets often occupy a low-dimensional manifold of these capabilities, necessitating a disentangled attribution approach for effective curation.
• The temporal self-influence of each intrinsic capability provides a quantitative signal for its learning difficulty and optimal progression, enabling a model-informed curriculum sequencing and balanced data allocation that optimizes VLM training.

Summary by alphaXiv.