Reproduce “Transmuting prompts into weights” on OpenResearch

About this paper

Hanna Mazzawi, Benoit Dherin, Michael Munn, Adrian Goldwaser, Michael Wunder, Javier Gonzalvo

Researchers at Google Research introduced "thought patching," a theoretical framework and algorithm that transmutes textual prompts into durable weight updates for Large Language Models, unifying existing control methods. This allows a patched model to replicate full-context performance, achieving 100% accuracy on algorithmic tasks after training with just 10 examples and reducing toxicity from 90.53% to 6.22% on a detoxification task.

Key insights

• The effectiveness of heuristic LLM control methods, such as averaging contrastive activations for steering and applying low-rank matrix updates for model editing, can be theoretically justified as natural outcomes of instruction encoding.
• Textual instructions can be precisely transmuted into static, token-independent "thought vectors" and "thought matrices" that permanently modify model weights.
• This approach enables data-efficient instruction encoding, allowing models to internalize and execute complex instructions with as few as 10 examples, replicating full-context performance.

Summary by alphaXiv.