Reproduce “Understanding Task Transfer in Vision-Language Models” on OpenResearch

About this paper

Bhuvan Sachdeva, Karan Uppal, Abhinav Java, Vineeth N. Balasubramanian

This research systematically analyzes how finetuning a Vision-Language Model on one visual perception task affects its zero-shot performance on others, introducing the Perfection Gap Factor (PGF) to quantify these effects. The study identified structured transfer patterns, with low-level and image-level tasks exhibiting high transferability, and found that larger models show increased positive transfer across tasks.

Key insights

• VLM finetuning induces structured and consistent patterns of positive and negative transfer across perception tasks, with low-level and image-level tasks showing the highest positive transferability and malleability.
• An increase in model size correlates with a proportional increase in average positive task transferability and malleability, suggesting greater representational capacity in larger models facilitates beneficial knowledge transfer.
• The Perfection Gap Factor (PGF) provides a practical tool for data selection, enabling improved finetuning efficiency and mitigation of negative transfer, with PGF-informed data mixtures sometimes surpassing direct finetuning on target tasks.

Summary by alphaXiv.