Reproduce “HyperTuning: Toward Adapting Large Language Models without Back-propagation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen
HyperTuning generates task-specific parameters from a few examples for a frozen downstream model, using a T5-based hypermodel to produce soft prefixes or LoRA parameters. On held-out P3 tasks, HyperT5-Prefix averages 54.6 accuracy, above a shared-prefix baseline at 49.7 but below T5-MTF-Few-shot at 55.4. This proof of concept does not eliminate back-propagation during hypermodel training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.