Reproduce “Beneath the Surface: Investigating LLMs' Capabilities for Communicating with Subtext” on OpenResearch

About this paper

Kabir Ahuja, Yuxuan Li, Andrew Kyle Lampinen

The authors evaluate subtext by testing whether models communicate intended meanings to the right audiences across four suites. In Visual Allusions, a game where clues should reach some but not all listeners, Gemini-2.5-Pro produced obvious clues 59.28% of the time. Shared context reduced obvious clues for some models, but models rarely inferred shared stories when this was unstated.

Key insights

• Current frontier LLMs exhibit a pervasive "literal bias," consistently struggling to generate and interpret nuanced, implied meanings in communicative tasks.
• LLMs showed improved selective communication when explicit common ground was provided, but faced difficulties in inferring common ground or effectively managing information asymmetry for multiple audiences.
• Interpretation of allegories by LLMs was influenced by paratextual factors like author/reader names and specified reader personas, aligning with literary theories.

Summary by alphaXiv.