Reproduce “Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nyamtulla Shaik, Fengjun Li, Bo Luo
The authors evaluate safety benchmarks for 26 open-source small language models, ranging from 124 million to 4 billion parameters. They analyze five suites, measuring ambiguous judge labels and recomputing rankings under alternative scoring policies. Ambiguity is associated with prompt and response features, and rankings shift on several suites. These results show sensitivity of benchmark scores, not models’ actual safety.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.