Reproduce “ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extraction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pengze Li, Jiaqi Liu, Junchi Yu, Lihao Liu, Mingyu Ding, Wanli Ouyang, Shixiang Tang, Xi Chen
Researchers introduced ARCHE, a task and benchmark designed to evaluate large language models' ability to extract structured, paradigm-based reasoning chains from scientific articles, representing them as Reasoning Logic Trees. Evaluations on 10 leading models revealed an average Entity Coverage of 51.4% but a low Reasoning Edge Accuracy of 28.3%, highlighting a significant gap in their capacity for logically sound scientific argumentation.
Key insights
- A fundamental gap exists between current LLM fluency and their ability to perform genuine, paradigm-grounded scientific reasoning, as demonstrated by low Reasoning Edge Accuracy.
- LLMs exhibit a trade-off between comprehensively covering scientific entities and maintaining the logical soundness of their reasoning steps.
- Models frequently struggle with adhering to strict structural and formal constraints of logical paradigms, producing numerous formatting errors or structural violations in reasoning chains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.