Reproduce “Better Security Bug Report Classification via Hyperparameter Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rui Shu, Tianpei Xia, Laurie Williams, Tim Menzies

The study evaluates whether tuning classifiers or balancing their training data is more effective for identifying security-related bug reports in five Chromium and Apache project datasets. SMOTE-based preprocessing, especially when its settings are optimized with SMOTUNED, generally improved recall more than classifier tuning alone, though it also increased false alarms and results varied by project.

Key insights

  • Across the tested conditions, optimizing imbalance-handling preprocessing was generally more effective than tuning classifiers alone: SMOTE or SMOTUNED ranked higher in 27 of 40 comparisons, tied in nine, and ranked lower in four.
  • Once preprocessing was optimized, FARSEC’s irrelevancy-pruning filters provided no additional benefit in these experiments; without optimization, filtering generally helped.
  • Higher recall comes with a workload trade-off: more ordinary reports may be flagged for security review, and the classifier is intended to support triage rather than determine whether a report describes an exploitable vulnerability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.