Reproduce “ZipIt! Merging Models from Different Tasks without Training” on OpenResearch

About this paper

George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, Judy Hoffman

Researchers at Georgia Tech developed "ZipIt!", a method to combine deep learning models trained on disparate tasks and with different initializations into a single multi-task model without additional training. The approach yielded up to 32.9% higher joint accuracy than prior merging techniques on category-split datasets and demonstrated robust performance when combining models for distinct tasks like classification and semantic segmentation.

Key insights

• Merging models from genuinely distinct tasks requires a generalized feature matching approach that goes beyond simple one-to-one neuron permutations, as demonstrated by the failure of prior methods in this setting.
• The ability to perform 'within-model' merges (leveraging intra-model redundancies) in addition to 'across-model' merges contributes to better performance and is theoretically shown to yield tighter bounds on loss increase.
• "Partial zipping" is crucial for effective multi-task merging, allowing models to share low-level, general features while preserving higher-level, task-specific features in separate unmerged heads.

Summary by alphaXiv.