Reproduce “DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations” on OpenResearch

About this paper

Krishna Sri Ipsit Mantri, Carola-Bibiane Schönlieb, Bruno Ribeiro, Chaim Baskin, Moshe Eliasof

DITASK introduces a parameter-efficient fine-tuning method for Vision Transformers in multi-task learning by applying neural diffeomorphic transformations to modulate singular values while preserving singular vectors of pre-trained weights. It achieves state-of-the-art performance on the PASCAL MTL dataset with a +3.22% average relative improvement over single-task baselines, using 75% fewer trainable parameters than leading alternatives.

Key insights

• Preserving the singular vectors of pre-trained weights, which define the feature space orientation, is crucial for effective and robust multi-task adaptation.
• Modulating only the singular values through neural diffeomorphic transformations allows for full-rank updates in terms of feature space coverage while maintaining parameter efficiency and preserving the learned feature hierarchy.
• Diffeomorphic transformations, being smooth and monotone, ensure that the relative importance and ordering of features (singular values) learned during pre-training are maintained during adaptation.

Summary by alphaXiv.