Reproduce “Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic Reasoning Limits” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiang Zhang, Juntai Cao, Jiaqi Wei, Yiwei Xu, Chenyu You
Large Language Models often underperform on symbolic and arithmetic reasoning despite Chain-of-Thought prompting; this study from UBC and collaborators demonstrates that prevalent subword tokenization schemes are a critical barrier, obscuring atomic reasoning units and leading to up to 80% accuracy degradation, while atomically-aligned input formats can unlock robust generalization even in smaller models.
Key insights
- Tokenization, particularly subword-based methods, is a critical and underappreciated bottleneck for LLM symbolic reasoning, as it merges or obscures the atomic units necessary for computation.
- The concept of 'Token Awareness' is introduced, highlighting that a model's ability to discern properties within tokens directly influences its capacity for logical alignment and generalization of symbolic procedures.
- The effectiveness of Chain-of-Thought in simulating recurrent computation is deeply conditioned by the fidelity of the token-to-state mapping, which can be severely degraded by unaligned tokenization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.