Reproduce “BanMANI: A Dataset to Identify Manipulated Social Media News in Bangla” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mahammed Kamruzzaman, Md. Minul Islam Shovon, Gene Louis Kim
BanMANI introduces a new dataset for identifying specific manipulated claims within Bangla social media news, developed using a semi-automatic methodology that combines large language model generation with human validation. The work provides a benchmark for this task, showing that current large language models struggle with precise identification of manipulated and original text spans.
Key insights
- The semi-automatic data collection methodology, combining LLM generation with human validation, offers an effective strategy for creating valuable linguistic resources in low-resource languages by addressing data scarcity and tool limitations.
- Exploratory analysis of the dataset revealed that political news constitutes the largest portion of manipulated articles, indicating a strong correlation between politically sensitive topics and information manipulation on social media.
- Large language models (LLMs) like fine-tuned GPT-3 (ada) still exhibit relatively low performance on precise span extraction tasks (identifying altered and original excerpts), suggesting inherent difficulties in the task itself rather than solely language-specific limitations, as performance was comparable to English benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.