Reproduce “Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhongxiang Sun, Qipeng Wang, Haoyu Wang, Xiao Zhang, Jun Xu

Researchers at Renmin University of China investigated reasoning hallucinations in large reasoning models (LRMs) from a mechanistic perspective, proposing a "Reasoning Score" to quantify step-level reasoning depth. They identified three distinct internal patterns of hallucination and developed a detection framework that outperformed baselines, alongside a mitigation strategy that improved reasoning accuracy and generalization.

Key insights

  • The Reasoning Score effectively differentiates between shallow pattern-matching and deeper reasoning engagement by observing stability or significant changes in late-layer internal representations.
  • Reasoning hallucinations are characterized by quantifiable internal model behaviors, including unstable reasoning trajectories, misguided attention to erroneous earlier steps, and 'spurious verification' where perceived deep reasoning steps coincide with internal instability.
  • Integrating step-level reasoning rewards into LRM training paradigms can encourage more meaningful intermediate reasoning, thereby improving factual accuracy and the reliability of the overall reasoning process, suggesting a path to more robust models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.