Reproduce “mSCoRe: a $M$ultilingual and Scalable Benchmark for $S$kill-based $Co$mmonsense $Re$asoning” on OpenResearch

About this paper

Nghia Trung Ngo, Franck Dernoncourt, Thien Huu Nguyen

mSCoRe, a multilingual and scalable benchmark, was developed to systematically evaluate Large Language Models' (LLMs) commonsense reasoning by dissecting processes into skill-labeled steps and dynamically scaling task complexity. Experiments revealed that LLMs struggle with higher complexity and cultural nuances, often over-relying on logical reasoning.

Key insights

• Current LLMs struggle to dynamically adapt their reasoning depth and diversify skill application, often showing an over-reliance on logical reasoning across all complexity levels, even when other skills are more appropriate.
• The fine-grained reasoning skill taxonomy within mSCoRe encourages models to explore a broader range of reasoning approaches, yielding better performance compared to less structured prompting methods like standard Chain-of-Thought.
• Specialized reasoning-reinforced training, while beneficial for formal tasks, can inadvertently reduce general commonsense reasoning ability in diverse linguistic contexts for larger models, suggesting a need for more balanced training.

Summary by alphaXiv.