Reproduce “A Comparative Study of Vulnerability Reporting by Software Composition Analysis Tools” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nasif Imtiaz, Seaver Thorne, Laurie Williams

An empirical comparison of nine software composition analysis tools shows that vulnerability reports can vary substantially depending on dependency-discovery methods, vulnerability databases, version mappings, and runtime or code-level analysis. Across Maven and npm components in OpenMRS, the tools reported widely different alert volumes and overlapping but incomplete sets of vulnerable dependencies, highlighting the limits of relying on a single SCA system.

Key insights

  • Tool output depends heavily on how dependencies are discovered: manifest, lock-file, source-code, binary, and runtime scanning can reveal different parts of an application’s dependency inventory.
  • Vulnerability databases and package mappings are not consistent; tools may disagree on affected versions, duplicate findings across related packages, or include proprietary and non-CVE advisories.
  • Code reachability and execution context can substantially change practical risk: Eclipse Steady found that most of its alerts concerned packages not identified as used by the application.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.