Reproduce “Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, David Scott Krueger
A mechanistic analysis of fine-tuning demonstrates that it primarily learns minimal "wrappers" on pre-existing capabilities rather than fundamentally altering or deleting them. This research, validated across synthetic and language modeling tasks, reveals that suppressed capabilities are recoverable, highlighting potential vulnerabilities in AI safety and alignment efforts.
Key insights
- Fine-tuning minimally transforms pretraining capabilities, typically by learning a "wrapper" as a localized modification on top of existing skills, rather than fundamentally altering or deleting them.
- These learned wrappers are localized and can be bypassed, meaning the original, seemingly suppressed capabilities often remain largely preserved and can be readily uncovered or revived.
- Reverse fine-tuning enables sample-efficient capability revival, demonstrating that capabilities are often dormant or inhibited rather than truly erased by fine-tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.