Reproduce “Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qifan Zhang, Jianhao Ruan, Aochuan Chen, Kang Zeng, Nuo Chen, Jing Tang, Jia Li
The Hong Kong University of Science and Technology (Guangzhou) researchers developed GRALGOBENCH, a benchmark using graph algorithm problems to evaluate Large Reasoning Models (LRMs). This benchmark demonstrates that LRMs show a notable decline in performance with longer input contexts and engage in ineffective self-verification during reasoning processes.
Key insights
- LRMs experience consistent performance degradation when presented with longer input contexts, whether from increased graph complexity or verbose textual descriptions.
- Primary failure modes in LRMs' graph reasoning include frequent coarse step-by-step execution errors, weak graph memorization, and the generation of excessive, non-contributory redundant content.
- The phenomenon of "over-thinking" in LRMs is driven mainly by self-verification attempts that rarely succeed in detecting or correcting prior reasoning errors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.