Reproduce “From Generalist to Specialist: Exploring CWE-Specific Vulnerability Detection” on OpenResearch

About this paper

Syafiq Al Atiiq, Christian Gehrmann, Kevin Dahlén, Karim Khalil

This research investigates moving from generalist to specialist vulnerability detection, proposing and evaluating CWE-specific and multi-class classification approaches using a fine-tuned LLM. The study demonstrates that specialized models perform better within their domain and that a multi-class approach improves overall detection compared to binary classification, despite persistent challenges with false positives due to data imbalance.

Key insights

• Training models on specific CWE types allows them to learn more targeted representations, leading to superior performance within their specialized domain compared to general binary classifiers.
• Specialized CWE models can detect vulnerabilities beyond their trained category, suggesting shared underlying characteristics among different CWEs, but this comes with a high false positive rate when evaluated broadly.
• Treating vulnerability detection as a multi-class problem, distinguishing between different CWE types and a non-vulnerable class, generally outperforms binary classification by capturing more fine-grained information.

Summary by alphaXiv.