Reproduce “LLMs in the SOC: An Empirical Study of Human-AI Collaboration in Security Operations Centres” on OpenResearch

About this paper

Ronal Singh, Shahroz Tariq, Fatemeh Jalalvand, Mohan Baruwal Chhetri, Surya Nepal, Cecile Paris, Martin Lochner

Researchers from Data61, CSIRO, and eSentire Inc. conducted a 10-month empirical study with 45 SOC analysts, revealing how Large Language Models function as on-demand cognitive aids for interpreting technical artifacts and refining communication in real-world security operations. The study found analysts primarily seek explanatory support, with 93% of queries aligning to the NICE Framework, and strongly prefer maintaining decision authority over receiving explicit recommendations.

Key insights

• LLMs serve as flexible, on-demand cognitive aids in SOCs, primarily supporting microtasks such as interpreting technical telemetry and refining communication, rather than complex, multi-step operations.
• SOC analysts show a strong preference for interpretive support and evidence over explicit recommendations from LLMs, actively maintaining decision authority in security investigations.
• The practical applications of LLMs in the SOC are highly relevant to established professional competencies, with 93% of analyst queries aligning with the NICE Framework.

Summary by alphaXiv.