Reproduce “On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks” on OpenResearch

About this paper

Stephen Obadinma, Xiaodan Zhu

Researchers at Queen's University's Ingenuity Labs demonstrated that large language models' verbal confidence is highly vulnerable to adversarial attacks, achieving up to a 29.5% reduction in expressed confidence and frequently altering answers. The study found that both subtle semantic perturbations and optimized trigger phrases can manipulate these confidence scores, and current defense mechanisms are largely ineffective against such threats.

Key insights

• Verbal confidence requires specific adversarial targeting; generic attacks that reduce accuracy only cause minor, incidental confidence drops.
• Attacks solely aimed at verbal confidence often achieve traditional adversarial objectives by also inducing high rates of answer changes (up to 100% answer flips).
• LLMs tend to exhibit overconfidence, rarely reporting very low certainty even under attack, and this distribution aligns with patterns of conversational human overconfidence.
• Existing defense mechanisms proved largely ineffective or even counterproductive against verbal confidence attacks, highlighting a critical security gap in current LLM robustness strategies.

Summary by alphaXiv.