Reproduce “CLOMO: Counterfactual Logical Modification with Large Language Models” on OpenResearch

About this paper

Yinya Huang, Ruixin Hong, Hongming Zhang, Wei Shao, Zhicheng Yang, Dong Yu, Changshui Zhang, Xiaodan Liang and 1 more

Researchers from City University of Hong Kong, Tsinghua University, and Tencent AI Lab introduce CLOMO, a new generative task and benchmark for evaluating large language models' counterfactual logical reasoning abilities. Their work demonstrates that fine-tuning significantly improves smaller models, while top-tier models like GPT-4o approach and even surpass human performance on this challenging task.

Key insights

• Counterfactual logical modification is a highly challenging task for both humans (SES 0.580) and LLMs, demanding deep logical understanding and precise text editing.
• Current pre-training data for smaller LLMs likely lacks sufficient counterfactual reasoning patterns, as fine-tuning with CLOMO yields substantial performance improvements (e.g., LLaMA 2-13B zero-shot SES from 0.045 to 0.580).
• "Weaken" and "Sufficient Assumption" logical relations are particularly difficult for LLMs, indicating specific areas where reasoning capabilities need further development.

Summary by alphaXiv.