Reproduce “ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Martina Miliani, Serena Auriemma, Alessandro Bondielli, Emmanuele Chersoni, Lucia Passaro, Irene Sucameli, Alessandro Lenci
This research introduces ExpliCa, a new dataset evaluated with crowdsourced human acceptability ratings, to assess Large Language Models' (LLMs) ability to perform explicit causal reasoning while distinguishing it from temporal relationships. The study reveals that current LLMs often struggle with this distinction, frequently confounding temporal precedence with causality, and exhibit a consistent gap between their internal linguistic competence and observable performance on prompting tasks.
Key insights
- A consistent gap exists between LLMs' internal linguistic competence (higher perplexity-based scores) and their observable performance in explicit causal reasoning tasks elicited via prompting.
- LLMs frequently confuse temporal precedence with causality, particularly in temporal iconic and causal anti-iconic contexts, indicating a less abstract understanding of these relations.
- The linguistic order of events significantly influences LLM performance, suggesting a reliance on surface-level features rather than robust, abstract relational logic for causal and temporal understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.