Reproduce “LLM Persuasion Is in the Eye of the Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kamile Dementaviciute, Julija Vaitonyte, Tijl De Bie
A comparison of nine automated LLM persuasion evaluations across 15 models finds that their rankings agree only weakly: the mean pairwise Spearman correlation was 0.252 across the eight reliable methods. Differences in task design, refusals, and general model capability all shape the scores, so a result from one evaluation should not be treated as a general measure of persuasiveness.
Key insights
- Agreement was limited even after accounting for measurement reliability, and the methods never produced identical model rankings.
- Refusals materially influenced results: excluding refusals raised mean agreement from 0.252 to 0.342, showing that scores can mix willingness to attempt a task with ability to persuade.
- Rational-persuasion evaluations tended to track general capability, while some manipulation-task evaluations did not; capability therefore contributed to the pattern of agreement between methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.