Reproduce “TOAST: Transfer Learning via Attention Steering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Baifeng Shi, Siyu Gai, Trevor Darrell, Xin Wang
The TOAST framework enables transfer learning by actively steering a frozen pre-trained model's internal attention towards task-relevant features. It achieves an average accuracy of 86.2% on FGVC classification, outperforms full fine-tuning on instruction-following language generation, and shows competitive results on VTAB-1k, often with significantly fewer tunable parameters.
Key insights
- Actively steering a pre-trained model's internal attention towards task-relevant features, rather than modifying core backbone parameters, serves as an effective transfer learning mechanism.
- Pre-trained models contain rich feature representations; effective transfer often lies in precisely selecting and highlighting these features for a specific task.
- Both token-wise and channel-wise attention within the feature selection module are crucial for adapting the model's focus, indicating the importance of both spatial and semantic guidance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.