Reproduce “Number Cookbook: Number Understanding of Language Models and How to Improve It” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haotong Yang, Yi Hu, Shijia Kang, Zhouchen Lin, Muhan Zhang
Researchers introduced the NUPA Test, a comprehensive benchmark designed to assess Large Language Models' (LLMs) intrinsic numerical understanding and processing abilities across various formats and tasks. The study reveals that despite advanced reasoning, LLMs struggle with basic numerical operations, especially with increasing number length and non-integer representations, while identifying effective pretraining strategies and limitations of finetuning architectural changes.
Key insights
- State-of-the-art LLMs possess surprising vulnerabilities in basic NUPA, with performance significantly declining as number length increases and for non-integer or digit-level operations.
- For enhancing NUPA and length generalization during pretraining, one-digit tokenizers and positional encodings like Alibi or NoPE are more effective than k-digit tokenizers and RoPE, which tend to encourage length-related shortcuts.
- Architectural or data format optimizations effective during pretraining for NUPA do not transfer well to finetuning existing LLMs, often disrupting their established knowledge and degrading performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.