Reproduce “Evaluating the Reliability of Multiple Large Language Models in Risk Assessment: A CIS Controls Based Approach” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gustavo Roberto Pinto, Arthur do Prado Labaki, Rodrigo Sanches Miani
Researchers at the Federal University of Uberlândia evaluated several large language models' ability to perform cybersecurity risk assessments against a set of CIS Controls-based scenarios, finding that LLMs consistently underestimated risks by an average of 1.66 points on a 0-10 scale compared to human cybersecurity professionals. The study highlights the critical need for human oversight in AI-driven risk assessment processes.
Key insights
- Large Language Models systematically underestimate cybersecurity risks compared to human experts, particularly when evaluated against senior and specialist professionals.
- Human expertise and professional background significantly influence risk perception, leading to higher and more consistent risk ratings among experienced professionals.
- LLMs should primarily serve as decision-support tools for initial assessments, requiring human validation, rather than operating as standalone, independent assessors in cybersecurity risk management.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.