Reproduce “Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zecheng Wang, Deyuan Liu, Chunshan Li, Yupeng Zhang, Zhengyun Zhao, Dianhui Chu, Bingning Wang, Dianbo Sui
Dynamic Entropy Fine-Tuning (DEFT) is a parameter-free objective proposed by researchers from Harbin Institute of Technology, WeChat, Tencent, and Tsinghua University, which unifies supervised fine-tuning (SFT) loss functions by dynamically modulating gradient influence based on the model's predictive uncertainty. This approach achieved consistent performance gains over standard Negative Log-Likelihood (NLL) and other baselines across various LLM capability regimes and demonstrated enhanced out-of-distribution generalization.
Key insights
- NLL's uniform gradient weighting during SFT creates a fundamental 'plasticity-stability dilemma' by both over-amplifying noisy signals for low-probability tokens and under-sharpening confident correct predictions.
- Any token-level SFT objective can be viewed as a 'trust gate' multiplied by a 'prediction error' in its gradient structure, revealing how objectives implicitly allocate learning signals based on model confidence.
- A dynamically adapting 'trust gate,' driven by the model's internal uncertainty (e.g., Rényi-2 entropy or a geometrically derived Cayley transform), can optimally balance exploring uncertain knowledge and exploiting confident knowledge by modulating gradient influence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.