Reproduce “IGC: Integrating a Gated Calculator into an LLM to Solve Arithmetic Tasks Reliably and Efficiently” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Florian Dietz, Dietrich Klakow

IGC adds an internal calculator to a pretrained instruction-finetuned Llama 3.1 8B, computing arithmetic inside the model rather than generating intermediate steps or calling an external tool. On BigBench Arithmetic, it reaches 0.99 overall accuracy and 0.99 on multiplication, versus 0.22 for a matched-parameter Llama adapter baseline trained on the same data. These results cover benchmark arithmetic, not word-problem reasoning.

Key insights

  • A 'left-aligned' numerical representation during tokenization, where the most significant digit is mapped to a fixed index, is crucial for the IGC to accurately process multi-digit numbers.
  • Integrating the IGC module at an early layer (e.g., layer 1) of the LLM proves most effective for enhancing arithmetic capabilities.
  • Hybrid architectures that introduce direct shortcuts bypassing the non-differentiable calculator lead to higher variance and lower performance, highlighting the importance of the calculator's inductive bias.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.