Reproduce “Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models” on OpenResearch

About this paper

Fei Yuan, Changjiang Gao, Yilei Tu, Yifeng Liu, Shujian Huang, Yu Qiao

Fine-tuning can be understood as spending a limited behavioral-drift budget, where the direction of parameter updates matters as much as their magnitude. Experiments on Qwen3 models show that layer-selective updates—especially split configurations such as b4t16—can improve scientific reasoning and multilingual translation while preserving general capabilities more effectively than full fine-tuning or KL regularization, and can provide stronger starting points for subsequent reinforcement learning.

Key insights

• Equal amounts of behavioral drift can lead to very different task outcomes, so KL magnitude alone is insufficient for evaluating fine-tuning strategies.
• Split layer updates often convert small drift budgets into better task gains than contiguous updates, while larger contiguous update regions can offer more expressivity when greater drift is acceptable.
• Effective update directions appear to form transferable families: the b4t16 configuration works across Qwen3-8B, Qwen3-14B, and InternS1-mini-8B, and also improves reinforcement-learning initialization.

Summary by alphaXiv.