Reproduce “Cross-Domain Evaluation of Transformer-Based Vulnerability Detection on Open & Industry Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Moritz Mock, Thomas Forrer, Barbara Russo
Researchers at Free University of Bozen-Bolzano and Würth Phoenix conducted a cross-domain evaluation of the CodeBERT model for vulnerability detection across open-source and proprietary industrial PHP codebases, identifying data balancing strategies that enhance recall on industrial data. They also developed AI-DO, a CI/CD-integrated tool, and assessed its perceived usefulness among IT professionals.
Key insights
- Undersampling strategies significantly improve recall in vulnerability detection, critical for minimizing false negatives, especially when adapting models across domains.
- Fine-tuning CodeBERT with a diverse, technology-agnostic open-source dataset and relative undersampling can lead to better recall on industrial code, despite a potential increase in false positives.
- The successful integration and adoption of AI-driven security tools in industry are heavily influenced by user perception, role-specific needs, and how well the tool explains its findings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.