Reproduce “Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset” on OpenResearch

About this paper

Saeid Naeini, Raeid Saqur, Mozhgan Saeidi, John Giorgi, Babak Taati

The Only Connect Wall dataset evaluates whether language models can form flexible cross-domain associations while resisting plausible but misleading clues. On 494 test puzzles, GPT-4 solved at most 7 complete walls and identified 272 groups, far below humans’ 285 solved walls and 1,405 correct groups; performance rose sharply when red herrings were removed, indicating strong sensitivity to distractor structure.

Key insights

• Broad semantic similarity is not enough for these puzzles: E5-base identified 89 groups and solved only one wall, while contextual embeddings sometimes amplified misleading associations.
• GPT-4 benefited from larger model capacity but remained well below humans; additional demonstrations were inconsistent for grouping and did not reliably teach a general solving strategy.
• The sharp improvement on simplified datasets shows that models are highly affected by ambiguity and distractors, though the ablations do not uniquely establish a human-like fixation mechanism.

Summary by alphaXiv.