Reproduce “Update Your Transformer to the Latest Release: Re-Basin of Task Vectors” on OpenResearch

About this paper

Filippo Rinaldi, Giacomo Capitani, Lorenzo Bonicelli, Donato Crisostomi, Federico Bolelli, Elisa Ficarra, Emanuele Rodolà, Simone Calderara and 1 more

TransFusion enables data-free and training-free transfer of fine-tuned knowledge, represented as task vectors, from an older Transformer base model to a newer version. The method achieves this by carefully aligning model weights and preserves functional equivalence, leading to improved zero-shot performance on downstream tasks, such as a 4.95% increase on EuroSAT and a 6.50% increase on QQP, while maintaining generalization capabilities.

Key insights

• A structured, two-level permutation approach (inter-head followed by intra-head alignment) can effectively re-base Transformer attention layers while provably preserving functional equivalence.
• Careful management of residual connections is critical to prevent representational mismatches when applying permutations across Transformer blocks, ensuring the coherence of the model's internal states.
• Fine-tuned knowledge, encoded as task vectors, can be successfully transferred to new foundation model releases in a data-free and training-free manner through permutation-based re-basing, extending model utility.

Summary by alphaXiv.