Reproduce “Transporting Task Vectors across Different Architectures without Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Angelo Porrello, Simone Calderara
THESEUS transfers a learned task update between different-width models by aligning their activation spaces from calibration examples and transporting the update through separate input and output maps. Its closed-form rule requires forward activations, not target-model backpropagation. On 8-Vision it reaches 68.35% average accuracy versus 58.76% for the target’s zero-shot baseline, using validation-set selection of update scale; functional matching is limited to aligned shared subspaces.
Key insights
- Task-specific knowledge can be effectively transferred across heterogeneous architectures by focusing on the functional impact of updates on intermediate representations, rather than direct parameter-space alignment.
- Geometric alignment of activation spaces via orthogonal Procrustes analysis can resolve the ambiguity of matching across different model widths, leading to a stable and norm-preserving transport rule.
- Accumulating activation statistics incrementally through cross-covariance matrices bounds memory usage, making the method scalable for large calibration datasets and diverse model sizes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.