Reproduce “InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jan Trienes, Sebastian Joseph, Jörg Schlötterer, Christin Seifert, Kyle Lo, Wei Xu, Byron C. Wallace, Junyi Jessy Li
This research introduces INFOLOSSQA, a framework for characterizing and recovering information loss in text simplification through reader-focused Question-Answer pairs. The work provides a linguist-curated dataset of 1,000 QA pairs derived from simplified medical abstracts, and evaluates automatic methods, finding that while LLMs generate accurate answers, consistently identifying human-perceived information loss remains challenging.
Key insights
- Oversimplification, where information becomes vague or imprecise, is more prevalent (74.1%) than complete deletion (25.9%) in medical text simplification.
- Human information needs regarding lost details are subjective, with low inter-annotator agreement on precise evidence spans, confirming varied perceptions of what constitutes salient missing information.
- While LLMs are adept at question generation and answer accuracy, their primary challenge is reliably detecting and precisely characterizing information loss from a human perspective, particularly distinguishing subtle oversimplifications and recalling all human-identified instances.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.