Reproduce “Navigating the OverKill in Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chenyu Shi, Xiao Wang, Qiming Ge, Songyang Gao, Xianjun Yang, Tao Gui, Qi Zhang, Xuanjing Huang and 2 more

A study by Fudan University identified that large language models exhibit "overkill," excessively refusing benign queries containing sensitive keywords due to shortcut learning and safety system prompts. They developed Self-Contrastive Decoding (Self-CD), a training-free method, which decreased refusal rates by an average of 20% while preserving safety performance.

Key insights

  • LLMs exhibit limited semantic understanding, often relying on "shortcut learning" by over-attending to individual harmful words rather than the full contextual meaning of a query.
  • Safety-emphasizing system prompts inadvertently exacerbate this shortcut behavior, increasing the model's reliance on superficial cues and leading to higher refusal rates.
  • The model's internal information flow and perplexity measurements can effectively diagnose and quantify this "over-attention" to harmful words and the impact of safety prompts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.