Reproduce “Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia” on OpenResearch

About this paper

Zhejian Zhou, Jiayu Wang, Dahua Lin, Kai Chen

This research systematically investigates how different numeral systems, implemented via tokenization, influence the arithmetic capabilities and scaling behavior of Large Language Models. The study reveals that a base 10 numeral system (single-digit tokenization) consistently offers superior data efficiency for training models on numerical tasks, particularly during from-scratch training, while also providing insights into the limitations of LLMs for out-of-domain length generalization in arithmetic.

Key insights

• The choice of numeral system (tokenization scheme) significantly impacts an LLM's data efficiency for arithmetic tasks, with base 10 (single-digit tokenization) offering a notable advantage for from-scratch training.
• Higher token frequencies inherent to base 10 systems appear to contribute to more robust learning of numerical operations.
• LLMs struggle with out-of-domain length extrapolation for precise arithmetic, often exhibiting truncated addition or fragmented understanding for longer numbers, though fine-tuned base 10 models show some ability to generalize carry operations.

Summary by alphaXiv.