Reproduce “A Study on Thinking Patterns of Large Reasoning Models in Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kevin Halim, Sin G. Teo, Ruitao Feng, Zhenpeng Chen, Yang Gu, Chong Wang, Yang Liu
A collaborative study from Nanyang Technological University and other institutions systematically analyzed the internal reasoning processes of Large Reasoning Models (LRMs) during code generation, developing a 15-action taxonomy and revealing distinct model reasoning patterns and their correlation with code correctness. The research found that generating unit tests strongly correlates with correct code and that context-aware prompting can improve LRM performance.
Key insights
- The developed taxonomy provides a structured framework for visualizing and understanding LRM reasoning in code generation, enabling clearer insights into their internal decision-making.
- LRMs exhibit distinct reasoning styles; Qwen3 models demonstrate an iterative, human-like coding workflow, while DeepSeek-R1 follows a more linear, waterfall-like approach, with iterative reasoning correlating positively with correctness.
- Unit Test Creation (UTC) by LRMs shows the strongest positive correlation with functional correctness, suggesting the potential for integrating test-driven development principles into LRM training.
- LRMs adapt their reasoning strategies based on task dependency levels, but their self-assertion and generated unit tests are not consistently reliable, highlighting the need for critical review of reasoning traces.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.