Reproduce “Loss Functions and Operators Generated by f-Divergences” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vincent Roulet, Tianlin Liu, Nino Vieillard, Michael E. Sander, Mathieu Blondel
Google DeepMind researchers introduce a framework for generating convex loss functions from f-divergences, generalizing the logistic loss and softargmax operator. The resulting "alpha-divergence" (with alpha=1.5) loss consistently matches or surpasses cross-entropy in tasks including ImageNet classification, language model pre-training, supervised fine-tuning, and distillation, while an efficient parallelizable bisection algorithm enables its practical use.
Key insights
- Fenchel-Young duality offers a principled method to derive a broad family of convex loss functions and corresponding f-softargmax operators from f-divergences.
- The alpha-divergence (specifically with alpha=1.5) yields a loss function that consistently performs competitively or better than cross-entropy across diverse tasks.
- An efficient and parallelizable bisection algorithm can compute the f-softargmax for any f-divergence, enabling practical deployment in deep learning systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.