Reproduce “Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Prisha Samadarshi, Mariam Mustafa, Anushka Kulkarni, Raven Rothkopf, Tuhin Chakrabarty, Smaranda Muresan

The authors introduce a benchmark of NYT Connections puzzles to test how language models group 16 words into four categories amid competing links. They evaluate five models with three solved examples and one-shot prompts, and classify the knowledge each category requires. Claude 3.5 Sonnet fully solved 18% of 438 games, showing limits in this model-and-prompt setup, not general reasoning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.