Reproduce “SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Paul Röttger, Fabio Pernisi, Bertie Vidgen, Dirk Hovy
The paper conducts the first systematic review of 144 open datasets for evaluating and improving Large Language Model safety, categorizing them by purpose, format, and creation methods, while also analyzing their practical usage in LLM releases and benchmarks. It finds an overwhelming English-centricity and a lack of naturalistic user interaction data, identifying a disconnect between dataset creation and real-world adoption in evaluations.
Key insights
- LLM safety dataset creation is experiencing unprecedented growth, with a recent shift towards broad and narrow safety tasks, including adversarial robustness and "jailbreaking" techniques.
- A significant gap exists in language diversity, with 78.5% of datasets exclusively in English, and a lack of datasets reflecting naturalistic user interactions, impacting ecological validity.
- Despite the abundance of open datasets, LLM model releases and popular benchmarks show limited and inconsistent utilization, often relying on proprietary data or a small, older subset of available resources.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.