Reproduce “UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoyu Lu, Yuqi Huo, Guoxing Yang, Zhiwu Lu, Wei Zhan, Masayoshi Tomizuka, Mingyu Ding
A unified parameter-efficient transfer learning method, UniAdapter, adapts large vision-language models for diverse cross-modal tasks, including image/video-text retrieval and question answering. It achieves performance comparable to or surpassing full fine-tuning methods while utilizing substantially fewer trainable parameters.
Key insights
- Sharing the down-projection layer across all adapter modules (visual, textual, cross-modal) while keeping up-projection layers modality-specific effectively reduces parameters and promotes knowledge sharing without sacrificing performance.
- Explicitly addressing language query integrity in cross-modal attention via a shared Query-Residual adapter improves performance without introducing additional parameters.
- A parameter-free frame-aware attention mechanism can efficiently handle temporal information and noise in video data, leading to performance gains on video tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.