Reproduce “Efficient numeracy in language models through single-token number embeddings” on OpenResearch

About this paper

Linus Kreitner, Paul Hager, Jonathan Mengedoht, Georgios Kaissis, Daniel Rueckert, Martin J. Menten

Researchers from Technical University of Munich and Imperial College London developed BitTokens, a single-token encoding strategy based on the IEEE 754 floating-point standard, to enhance Large Language Models' intrinsic numerical processing. This method enables small models to achieve near-perfect accuracy on basic arithmetic operations with high token efficiency, significantly outperforming prior single-token encodings and reducing reliance on verbose reasoning chains.

Key insights

• Frontier LLMs exhibit significant reliance on thousands of reasoning tokens (5,000-30,000) for difficult arithmetic tasks, indicating poor intrinsic numerical capabilities and high inefficiency.
• A structured binary encoding like IEEE 754, as used in BitTokens, provides a more suitable representation for neural networks to learn direct arithmetic algorithms compared to magnitude-based or sinusoidal encodings.
• While single-token encodings like BitTokens greatly improve efficiency and single-step arithmetic, handling complex multi-step calculations in a single pass remains a challenge, suggesting the potential benefit of combining efficient encodings with targeted reasoning for problem decomposition.

Summary by alphaXiv.