Reproduce “HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Samir Abdaljalil, Hasan Kurban, Erchin Serpedin
Researchers at Texas A&M University and Hamad Bin Khalifa University introduced HalluVerse25, a fine-grained multilingual benchmark dataset in English, Arabic, and Turkish, to analyze and detect different types of Large Language Model (LLM) hallucinations. The dataset and subsequent evaluations reveal that LLMs, including state-of-the-art models like GPT-4o, consistently struggle with identifying complex sentence-level factual errors more than entity or relation-level inaccuracies.
Key insights
- Multilingual, fine-grained annotation is crucial for a nuanced understanding of LLM hallucination types and their manifestation across diverse languages.
- Sentence-level hallucinations consistently pose a significantly greater challenge for current LLMs to detect compared to entity or relation-level errors, often leading to misclassification.
- The constructed HalluVerse25 dataset provides a valuable benchmark for identifying specific weaknesses in LLM factual consistency beyond English, guiding future model development.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.