Reproduce “Prompt Tuning for Generative Multimodal Pretrained Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Yang, Junyang Lin, An Yang, Peng Wang, Chang Zhou, Hongxia Yang
DAMO Academy researchers demonstrate that prompt tuning effectively adapts large generative multimodal models like OFA to various downstream tasks. For large models (~470M parameters), prompt tuning achieves performance comparable to full finetuning while updating only about 1% of parameters, and also exhibits enhanced robustness to adversarial attacks.
Key insights
- Prompt tuning's performance closely matches full finetuning only for larger pretrained models, suggesting its effectiveness scales with model size.
- Freezing the majority of model parameters via prompt tuning significantly improves robustness against adversarial attacks compared to finetuning.
- Optimal prompt tuning configuration involves inserting prompts into both the encoder and decoder, with a recommended prompt length of around 64 tokens for best overall performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.