Reproduce “AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haokai Ma, Javier Yong, Yunshan Ma, Kuei Chen, Anis Yusof, Zhenkai Liang, Ee-Chien Chang
AttackSeqBench evaluates whether models reconstruct the order and relationships among adversary actions in cyber threat intelligence reports. The authors extract tactics, techniques, and procedures from 408 reports and test models with multiple-choice and yes/no questions. A comparison gives Qwen2.5-32B 0.7267 accuracy versus 0.6569 for R1 (Qwen2.5-32B); reasoning models do not consistently outperform others, and results are limited to this constructed benchmark.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.