Reproduce “Multitask-based Evaluation of Open-Source LLM on Software Vulnerability” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xin Yin, Chao Ni, Shaohua Wang

Researchers at Zhejiang University and Central University of Finance and Economics systematically evaluated open-source Large Language Models across four software vulnerability tasks: detection, assessment, location, and description. The study found that while fine-tuning substantially improves LLM performance, specialized pre-trained models often surpass LLMs in detection and description, whereas fine-tuned code-related LLMs excel in vulnerability assessment and location.

Key insights

  • Fine-tuning is crucial for LLMs to achieve competitive performance in software vulnerability tasks, often doubling F1-scores compared to few-shot settings, and domain-specific fine-tuning is necessary.
  • The effectiveness of LLMs varies significantly across different vulnerability tasks; specialized pre-trained LMs can be more efficient for tasks like detection and description, while fine-tuned code-related LLMs are more beneficial for assessment and location.
  • LLM performance is sensitive to input preprocessing (e.g., code normalization) and can decline with excessively long input sequences, highlighting practical limitations for certain models in real-world scenarios.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.