Reproduce “How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guhao Feng, Kai Yang, Yuntian Gu, Xinyue Ai, Shengjie Luo, Jiacheng Sun, Di He, Zhenguo Li and 1 more
This research provides a theoretical framework explaining how numerical precision fundamentally impacts Large Language Models' arithmetical reasoning, demonstrating that lower precision leads to severe computational limitations for complex tasks like iterated addition and multiplication. The work shows that while basic addition is less sensitive, operations like iterated addition and multiplication become practically infeasible for models constrained to constant-precision internal representations.
Key insights
- Low-precision Transformers (constant depth, polynomial hidden dimensions) cannot efficiently solve iterated addition or integer multiplication, requiring super-polynomial model sizes due to their AC0-bounded expressiveness.
- Logarithmic-precision Transformers, which can store O(log n) bits per neuron, efficiently solve these arithmetic tasks with significantly smaller model sizes, demonstrating capabilities equivalent to TC0 circuits.
- The ability to store and manipulate intermediate carry values is critical for complex arithmetic, and this capability is directly tied to the numerical precision of the Transformer's internal representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.