Reproduce “Missci: Reconstructing Fallacies in Misrepresented Science” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Max Glockner, Yufang Hou, Preslav Nakov, Iryna Gurevych
The MISSCI framework from Max Glockner, Yufang Hou, Preslav Nakov, and Iryna Gurevych analyzes how scientific research is misrepresented in online claims by reconstructing the implicit, fallacious logical arguments. This approach not only identifies an inaccurate claim but also explains the specific reasoning error, and experiments show that large language models like GPT-4 can identify at least one fallacy in 57% of arguments, with performance boosted by providing fallacy definitions.
Key insights
- GPT-4 significantly outperforms LLaMA 2 in identifying fallacies, correctly detecting at least one fallacy in 57% of arguments.
- Providing fallacy definitions to large language models consistently improves their performance in fallacy classification and premise generation.
- While models can generate plausible fallacious premises, they often struggle with internal consistency, meaning the generated premise may not align with the fallacy class they predict.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.