Reproduce “Chimera: Harnessing Multi-Agent LLMs for Automatic Insider Threat Simulation” on OpenResearch

About this paper

Jiongchi Yu, Xiaofei Xie, Qiang Hu, Yuhan Ma, Ziming Zhao

Researchers from Singapore Management University, Tianjin University, and Zhejiang University introduce Chimera, an LLM-based multi-agent framework that automatically simulates diverse organizational operations and insider threats to generate high-fidelity, multi-modal security logs. The resulting ChimeraLog dataset achieves an expert-rated realism score of 4.20, comparable to real-world datasets, and presents a more challenging benchmark for ITD models, leading to up to a 20% drop in F1-score for existing methods compared to synthetic benchmarks.

Key insights

• Multi-agent LLM systems are effective for simulating complex enterprise scenarios and generating high-fidelity, difficult-to-collect synthetic datasets for cybersecurity research.
• The dynamic log generation capability can help mitigate performance degradation in ITD models caused by data distribution shifts in evolving enterprise environments by facilitating continuous retraining.
• While LLMs significantly enhance simulation, full automation of threat environment generation and adaptive, complex attack workflow definition remains an area for further development.

Summary by alphaXiv.