Reproduce “LLMs May Perform MCQA by Selecting the Least Incorrect Option” on OpenResearch

About this paper

Haochun Wang, Sendong Zhao, Zewen Qiang, Nuwa Xi, Bing Qin, Ting Liu

Large Language Models (LLMs) often perform Multiple Choice Question Answering (MCQA) by selecting the "least incorrect" option rather than identifying a uniquely correct one, even when exhibiting consistent performance across option orderings. Experiments show LLMs frequently misclassify incorrect statements as correct and struggle to identify questions with no correct answer, leading to the proposal of MCQA+, an augmented evaluation benchmark, which causes significant accuracy drops for all tested LLMs.

Key insights

• LLMs' consistent correctness on traditional MCQA does not imply genuine understanding; they frequently misclassify incorrect options as partially correct when presented in different formats.
• LLMs often attribute substantial partial correctness to incorrect options, quantitatively evidenced by high confidence scores for "most plausible incorrect" choices and struggles with scenarios lacking a clear, single correct answer.
• The "least incorrect" phenomenon suggests LLMs operate on a probabilistic continuum of correctness, challenging the fundamental assumption that MCQA performance reflects a binary understanding of right and wrong.

Summary by alphaXiv.