Reproduce “Argument Reconstruction as Supervision for Critical Thinking in LLMs” on OpenResearch

About this paper

Hyun Ryu, Gyouk Chu, Gregor Betz, Eunho Yang, Carolyn Rose, Sean Welleck

Researchers from Carnegie Mellon University, KAIST, and KIT developed a Generalized Automatic Argument Reconstruction (GAAR) engine and the Arguinas dataset to train large language models (LLMs) in argument reconstruction. This training consistently improved LLM performance across six of seven diverse critical thinking tasks, including argument quality evaluation and reasoning, with observed F1 score increases of up to 51.3% on specific benchmarks and enhanced data efficiency in low-resource settings.

Key insights

• Explicitly training LLMs on argument reconstruction using a high-quality, generalized dataset like Arguinas significantly enhances their performance on various downstream critical thinking tasks, demonstrating that this skill is transferable and foundational.
• The quality of argument reconstruction data is crucial; higher fidelity reconstructions (e.g., from GAAR with faithfulness feedback) lead to greater performance gains in downstream tasks.
• Specific characteristics of argument data, such as the inclusion of fallacious arguments (detrimental), the author of the argument (human-written is better), and the ratio of implicit premises (task-dependent), have differential impacts on LLM learning for critical thinking.

Summary by alphaXiv.