Reproduce “Reasoner Outperforms: Generative Stance Detection with Rationalization for Social Media” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaqing Yuan, Ruijie Xi, Munindar P. Singh
Researchers at North Carolina State University developed a generative stance detection framework that provides explicit rationales for predictions on social media content. This approach distills LLM-generated rationales into smaller language models, achieving up to a 79.72% macro-average F1-score and outperforming a zero-shot LLM baseline.
Key insights
- Multitask Learning (MTL) proved superior for distilling rationales into SLMs, allowing flexible and independent generation of both stance and rationales, fostering stronger connections between them.
- SLMs struggled with the Single-Task Chain-of-Thought (ST-CoT) approach, indicating that forcing sequential rationale generation before prediction is less effective with limited training data.
- Instruction-tuned models (FlanT5) exhibited better baseline performance and inherent reasoning capabilities than non-instruction-tuned T5 models, enhancing overall stance classification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.