Reproduce “The Use of Large Language Models (LLM) for Cyber Threat Intelligence (CTI) in Cybercrime Forums” on OpenResearch

About this paper

Vanessa Clairoux-Trepanier, Isa-May Beauchamp, Estelle Ruellan, Masarah Paquet-Clouston, Serge-Olivier Paquette, Eric Clay

Researchers at Universit R de Montr R eal, Complexity Science Hub, and Flare Systems evaluated an LLM system based on OpenAI's GPT-3.5-turbo for extracting Cyber Threat Intelligence (CTI) from cybercrime forum discussions without specific training. The system achieved an average accuracy of 96.23% across ten CTI variables, with an average precision of 90% and recall of 88.2% for binary variables.

Key insights

• LLMs can reliably summarize conversations and extract specific CTI variables from complex cybercrime forum data with high accuracy in a zero-shot setting, validating their utility for CTI automation.
• The performance of LLMs is sensitive to prompt design, particularly in distinguishing temporal context (e.g., active vs. past events) and handling vague concepts without explicit definitions.
• Data processing strategies, such as how conversations are chunked and the incorporation of crucial contextual elements like thread titles, significantly impact the LLM's ability to capture complete and accurate information.

Summary by alphaXiv.