Reproduce “Overinformative Question Answering by Humans and Machines” on OpenResearch

About this paper

Polina Tsvilodub, Michael Franke, Robert D. Hawkins, Noah D. Goodman

Researchers investigated how humans provide overinformative answers in conversational contexts, selecting information relevant to a questioner's practical goals, and then evaluated large language models' ability to replicate this context-sensitive pragmatic behavior. Findings indicate that while humans flexibly adapt responses based on functional relevance, most LLMs struggle with this without specific prompting, though GPT-3 can approximate human-like patterns when provided with one-shot examples.

Key insights

• Human overinformativity is driven by reasoning about the questioner's underlying action goals, leading to the selection of functionally relevant information that is flexibly adjusted based on specific contextual cues.
• Most general-purpose neural language models struggle to spontaneously identify and prioritize contextually relevant information in overinformative responses, often defaulting to providing exhaustive or less precise details.
• The pragmatic capabilities of advanced LLMs like GPT-3 are highly sensitive to prompt structure; providing one-shot examples, particularly explicit example answers, is more critical than mere explanations for eliciting human-like, context-aware responses.

Summary by alphaXiv.