Reproduce “Tuning Language Models by Proxy” on OpenResearch

About this paper

Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, Noah A. Smith

Researchers at the University of Washington and Allen Institute for AI developed 'proxy-tuning,' a decoding-time algorithm that enables the customization of large language models, including black-box models, without requiring access to their internal weights. This method uses logit arithmetic from a small, task-specific 'expert' model to guide a larger base model, achieving performance comparable to or exceeding direct fine-tuning across instruction-following, domain adaptation, and task-specific finetuning while also improving factual accuracy on benchmarks like TruthfulQA.

Key insights

• Proxy-tuning effectively transfers desired behaviors from small, fine-tuned expert models to larger base models without direct weight access, preserving the large model's general knowledge.
• The method mitigates the 'alignment tax' often associated with direct instruction tuning, enabling proxy-tuned models to sometimes surpass directly tuned models in factual accuracy (e.g., on TruthfulQA).
• Decoding-time logit arithmetic provides fine-grained control over generation style and can promote tokens that are extremely unlikely in the base model's original distribution.

Summary by alphaXiv.