Reproduce “Com$^2$: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models” on OpenResearch

About this paper

Kai Xiong, Xiao Ding, Yixin Cao, Yuxiong Yan, Li Du, Yufei Zhang, Jinglong Gao, Jiaqian Liu and 2 more

Researchers from Harbin Institute of Technology introduce Com², a new benchmark for evaluating Large Language Models' (LLMs) complex commonsense reasoning by structuring implicit, multi-step knowledge via causal event graphs. The benchmark reveals LLMs' struggles with reasoning depth and breadth in complex scenarios, demonstrating that post-training and explicit "slow thinking" guidance consistently improve their performance.

Key insights

• LLMs exhibit significant limitations in reasoning depth and breadth for complex commonsense tasks, particularly in handling transitive causal problems (Transition) and uncommon or sudden events (Intervention).
• Post-training LLMs on complex commonsense data significantly improves performance on in-distribution tasks and shows modest gains on out-of-distribution data.
• Explicit "slow thinking" guidance markedly enhances LLM performance across both Com²-main and Com²-hard, highlighting the value of structured reasoning steps.
• The test-time scaling law (generating more tokens for Chain-of-Thought) does not consistently improve complex commonsense reasoning, suggesting that mere length of reasoning is less critical than its quality and guidance.

Summary by alphaXiv.