Reproduce “Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jonas Zausinger, Lars Pennig, Anamarija Kozina, Sean Sdahl, Julian Sikora, Adrian Dendorfer, Timofey Kuznetsov, Mohamad Hagog and 8 more
The Number Token Loss (NTL) offers a training-free, regression-like loss function to enhance language models' intrinsic numerical reasoning by directly modeling numerical proximity at the token level. This method robustly improves arithmetic capabilities and general numerical accuracy across diverse model scales and architectures, evidenced by a 0.75 accuracy on arithmetic interpolation tasks with T5-Base and a 17.7% top-1 accuracy on GSM8k with T5-3B.
Key insights
- A direct modification to the loss function at the token level can fundamentally improve an LM's inductive bias for numerical proximity, bypassing complex architectural changes or external tools.
- The Wasserstein-1 distance (NTL-WAS) provides a robust and computationally efficient measure to quantify numerical proximity between predicted and true distributions over number tokens.
- NTL enables LMs with a standard token-head to achieve performance comparable to models with dedicated regression heads on numerical prediction tasks, without sacrificing general language modeling capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.