Reproduce “Over-Reasoning and Redundant Calculation of Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Cheng-Han Chiang, Hung-yi Lee

Researchers at National Taiwan University investigated over-reasoning in large language models, establishing that LLMs frequently produce redundant step-by-step calculations that reduce answer accuracy. Their work, which included the creation of the GSM8K-Zero dataset, further identified that reward models used in Reinforcement Learning with Human Feedback often prefer verbose outputs, even if unnecessary or incorrect.

Key insights

  • LLMs exhibit a strong tendency towards over-reasoning, frequently generating unnecessary step-by-step calculations even for questions that can be answered by direct extraction.
  • Redundant reasoning often correlates with a decrease in the accuracy of LLM responses, indicating that verbosity can detract from correctness.
  • Reward models guiding LLM training via Reinforcement Learning with Human Feedback (RLHF) appear to be biased towards longer, more detailed outputs, potentially encouraging over-reasoning regardless of its necessity or correctness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.