Reproduce “AQuA -- Combining Experts' and Non-Experts' Views To Assess Deliberation Quality in Online Discussions Using LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maike Behrendt, Stefan Sylvius Wagner, Marc Ziegele, Lena Wilms, Anke Stoll, Dominique Heinbach, Stefan Harmeling
AQuA introduces an interpretable, normalized score that automatically assesses the deliberative quality of individual comments in online discussions. It integrates predictions from 20 fine-tuned language model adapters for expert-defined criteria, weighted by their correlation with non-experts' perceptions, and achieves an F1 score of 81.73 in predicting comment constructiveness on external English datasets.
Key insights
- Rationality criteria, particularly "Solution Proposals" (0.395) and "Justification" (0.290), are the strongest positive indicators for non-experts' perception of overall deliberative quality.
- Negative indicators like "Sarcasm" (-0.152) and unsupported "Opinion" (-0.111) are perceived by non-experts as detrimental to discussion quality, aligning with expectations for constructive discourse.
- The AQuA score's interpretability, derived from its weighted additive structure, allows users to understand the specific contributions of individual deliberative aspects to a comment's overall quality assessment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.