Reproduce “LLM-based Detection of Manipulative Political Narratives” on OpenResearch

About this paper

Sinclair Schneider, Florian Steuber, Gabi Dreo Rodosek

Researchers at the University of the Bundeswehr Munich developed a multi-stage, LLM-driven pipeline to detect and structure manipulative political narratives within over 1.2 million unfiltered social media posts. The system successfully identified 41 distinct manipulative narrative clusters and accurately extracted their core claims, targeted adversaries, and manipulative angles.

Key insights

• Large Language Models, guided by prompt-based reasoning with explicit FIMI characteristics and few-shot examples, can effectively differentiate subtle manipulative narratives from legitimate political critique in noisy social media data.
• Adapting topic modeling pipelines by generating 'intent-driven' embeddings enables the discovery of underlying manipulative strategic narratives rather than merely topical content.
• A synergistic pipeline combining high-recall prompt-based filtering with density-based clustering effectively handles false positives by categorizing isolated, non-coordinated content as noise.
• Large reasoning models can integrate trained open-source intelligence knowledge beyond explicit prompt instructions, as demonstrated by the recognition of 'rtde.media' as a 'Russia Today' proxy.

Summary by alphaXiv.