Reproduce “Reasoning Models are Test Exploiters: Rethinking Multiple-Choice” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Narun Raman, Taylor Lundy, Kevin Leyton-Brown

Researchers at the University of British Columbia quantify how large language models (LLMs) exploit multiple-choice question answering (MCQA) formats, revealing that performance gains often stem from leveraging answer options rather than pure reasoning. The study shows that reasoning models exhibit substantial accuracy increases of 30-40 percentage points when options are presented before Chain-of-Thought, and that introducing a "None of the Above" option significantly mitigates this exploitation.

Key insights

  • Larger, more performant reasoning models demonstrate substantial exploitation, achieving 30-40 percentage points higher accuracy when options are available during Chain-of-Thought (QMC-CoT) compared to free-form reasoning (Q-CoT).
  • Exploitation manifests in different forms, including 'MC-only exploitation' where models perform better than random guessing from options alone, and 'QMC-based exploitation' where options are leveraged even with the question present.
  • The inclusion of a 'None of the Above' (NOTA) option significantly reduces exploitative behaviors, narrowing the performance gap between reasoning and non-reasoning models and disrupting 'second-pass shortcuts'.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.