Reproduce “Generating Verifiable Chain of Thoughts from Exection-Traces” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shailja Thakur, Vaibhav Saxena, Rohan Kulkarni, Shivdeep Singh, Parameswaran Selvam, Hima Patel, Hiroshi Kanayama
IBM Research developed a scalable pipeline to create execution-trace-verified Chain-of-Thought (CoT) rationales, grounding each natural language reasoning step in actual program execution. This approach leads to substantial improvements in LLM performance on code reasoning tasks, including gains up to +26.6% Pass@1 on LiveCodeBench-Exec and +19.5% on HumanEval, by training models on factually accurate, bi-directional explanations.
Key insights
- Data quality, specifically execution-grounded verifiability, is more critical than data quantity for enhancing LLM code reasoning capabilities and mitigating hallucination.
- Training LLMs with bi-directional CoT rationales (both forward and backward reasoning) provides complementary gains in code understanding and reasoning without compromising execution performance.
- Systematic, step-by-step verification of natural language rationales against actual program execution traces is essential for producing reliable CoT and improving LLM performance across coding tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.