Reproduce “Fooling LLM graders into giving better grades through neural activity guided adversarial prompting” on OpenResearch

About this paper

Atsushi Yamamura, Surya Ganguli

Researchers at Stanford University's Department of Applied Physics developed a systematic, neural activity-guided method to create adversarial prompts that cause Large Language Model essay graders to assign higher scores. They found that a hidden bias, dubbed the 'nonsense user effect' stemming from chat template design, could be exploited to manipulate LLM evaluations and then effectively mitigated by altering the 'user' token in fine-tuning templates.

Key insights

• Large Language Models form implicit preliminary score judgments early in their processing, identifiable through linear readouts from their hidden layers.
• A hidden bias, termed the "nonsense user effect," exists where the presence of the "user" token from chat templates can significantly distort LLM evaluative decisions.
• This bias originates from the structure of commonly used chat templates during supervised fine-tuning, influencing how LLMs interpret subsequent input.

Summary by alphaXiv.