Reproduce “When Contextual Inference Fails: Cancelability in Interactive Instruction Following” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Natalia Bila, Kata Naszádi, Alexandra Mayn, Christof Monz
Researchers from Saarland University and the University of Amsterdam developed "Build What I Mean," an interactive block-building benchmark, to evaluate how large language models distinguish literal meaning from contextual inference. Their findings reveal a dissociation where models may recognize pragmatic unreliability through confidence shifts but often fail to translate this awareness into efficient, adaptive information-seeking strategies.
Key insights
- LLMs exhibited a dissociation between their internal sensitivity to pragmatic unreliability (shown by shifts in confidence ratings) and their external, adaptive action selection (clarification questions).
- Current end-to-end LLMs may not fully possess a human-like "two-tier" system for processing meaning, particularly regarding the dynamic cancellation of pragmatic inferences and strategic adaptation to partner behavior.
- Models often either avoided clarification entirely, leading to suboptimal accuracy, or engaged in partner-blind over-clarification, failing to strategically balance communication costs and task success based on partner reliability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.