Reproduce “Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siqi Zeng, Yifei He, Meitong Liu, Weiqiu You, Yifan Hao, Yao-Hung Hubert Tsai, Makoto Yamada, Han Zhao
The authors introduce a learned autoencoder that compresses model-sized task vectors into fewer reusable bases for model merging. Its softmax encoder forms each basis as a nonnegative mixture of task vectors. On a 12-task RoBERTa-base language benchmark, three bases, 25% of the task vectors, retain 97% of the full-vector Localize-and-Stitch score; this finding is specific to that benchmark.
Key insights
- Standard dimensionality reduction techniques like PCA are ill-suited for task arithmetic due to their basis vectors not inherently supporting non-negative scaling coefficients or preserving task semantic interpretability.
- Constructing basis vectors as convex combinations of original task vectors (via a softmax-activated autoencoder) is crucial for maintaining functional properties and interpretability in task arithmetic.
- Theoretical analysis indicates that the proposed basis addition and negation methods can maintain generalization performance and unlearning capabilities, with performance bounds tied to the reconstruction quality of the basis set.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.