Reproduce “When the Text Is Silent: Memory Shortcuts and the Validity of LLM Annotation under Evidence Insufficiency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenchang Wang, Yue Song
The authors test whether language models use remembered facts about named people when annotating text that omits relevant evidence. In a counterfactual audit of 995 public figures and matched names, GPT-5.6 Luna produced a 17.0-percentage-point occupation shortcut contrast; tested models did not show this behavior when text conflicted with memory. Results are limited to stylized narratives and three commercial models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.