Reproduce “HarmProfile: Characterizing Harmful Distributions in Frontier LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhouyuan Ma, Yutao Wu, Hanxun Huang, Xiang Zheng, Xiao Liu, Yixin Cao, Zuxuan Wu, Xingjun Ma and 1 more
HarmProfile introduces a framework and benchmark dataset for evaluating the safety of frontier large language models (LLMs) by shifting from binary attack outcomes to a content-centric distributional analysis of harmful outputs. The framework characterizes models' specific 'harmful profiles' based on the content, severity, and diversity of their safety failures, revealing that harmfulness and diversity generally increase with model capability across 23 evaluated LLMs.
Key insights
- Despite extensive safety alignment, frontier LLMs retain the capacity for generating diverse and severe harmful content, with specific 'harmful distributions' rather than uniform failure patterns.
- Harmfulness and diversity of generated content exhibit positive associations with general model capability (e.g., LMArena Elo scores), suggesting that increased capability may amplify safety risks.
- Models possess latent harmful knowledge that extends beyond what zero-shot autonomous elicitation methods can readily expose, implying an 'elicitation boundary' rather than a complete 'knowledge boundary'.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.