Reproduce “Long-Form Information Alignment Evaluation Beyond Atomic Facts” on OpenResearch

About this paper

Danna Zheng, Mirella Lapata, Jeff Z. Pan

Researchers at the University of Edinburgh introduce MONTAGELIE, a benchmark designed to expose how large language models can create deceptive narratives by subtly reordering truthful statements. They also present DOVESCORE, an evaluation framework that significantly improves in identifying these "montage-style lies" by assessing both factual accuracy and event chronology.

Key insights

• Existing information alignment evaluators, including powerful LLM-as-evaluator methods and fine-grained frameworks like FactScore, are highly susceptible to "montage-style lies," achieving low detection accuracy.
• The inability to effectively model and verify inter-fact dependencies, particularly event chronology, is a fundamental limitation of current fine-grained evaluation frameworks.
• Detecting subtle causal distortions introduced by event reordering requires significant reasoning capacity, with larger LLMs showing better but still inadequate performance on this challenging task.

Summary by alphaXiv.