Reproduce “Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jungyong Son, Jinwook Jung, Minhee Park, Sungyong Baik
The authors present BiCo, a method for reusing fine-tuned updates across related models. It maps both layer input activations and output-side gradients between source and target models, then applies the transferred update to the target. In an eight-task vision comparison, BiCo reaches 75.51% average accuracy at K=1, above competing methods but below direct fine-tuning, in the tested related model families.
Key insights
- Task vectors represent an accumulation of bilinear interactions between input-side activations and output-side gradients over the fine-tuning process.
- Effective task-vector transfer necessitates aligning both the input-side activation space and the output-side gradient space, as these components are intrinsically coupled.
- Orthogonal Procrustes alignment, applied to pre-trained activations and local output gradients from a small calibration set, can robustly achieve this dual-space alignment without further training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.