Reproduce “Logical forms complement probability in understanding language model (and human) performance” on OpenResearch

About this paper

Yixuan Wang, Freda Shi

Researchers from the University of Chicago, University of Waterloo, and Vector Institute investigated how propositional and alethic modal logic forms influence large language model (LLM) reasoning. They found that LLM performance is significantly affected by specific logical structures and modalities, with LLMs consistently performing better on tasks involving possibility and struggling with necessity, often exhibiting biases not present in human reasoning.

Key insights

• Logical forms, including specific argument structures and modalities, are crucial, complementary factors in predicting and explaining LLM logical reasoning performance, beyond input perplexity.
• LLMs consistently struggle with the necessity modality, modus tollens, and affirming the consequent, often exhibiting specific biases like a systematic 'rejection bias' for necessity.
• While LLMs and humans share some reasoning patterns, significant discrepancies exist, such as the LLMs' strong rejection bias for the necessity modality which was not observed in human participants.

Summary by alphaXiv.