Reproduce “Using LLMs in Software Requirements Specifications: An Empirical Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Madhava Krishna, Bhagesh Gaur, Arsh Verma, Pankaj Jalote
Researchers at Krishna et al. conducted a systematic empirical evaluation of Large Language Models (LLMs) like GPT-4 and CodeLlama-34b for generating, validating, and correcting Software Requirements Specification (SRS) documents. The study found that LLMs could achieve up to 47x speedup in SRS generation compared to human efforts, demonstrating strong capabilities in certain requirement types but showing limitations in complex technical specifications such as design constraints and external interfaces.
Key insights
- LLMs like GPT-4 and CodeLlama can generate structured SRS documents that often match or exceed human benchmarks in specific quality dimensions like completeness, consistency, and security requirements.
- GPT-4 demonstrated strong capabilities in validating existing requirements, providing detailed and actionable feedback aligning with human expert judgment, and successfully correcting identified issues.
- While LLMs excel in certain areas, they show limitations in handling complex technical specifications such as design constraints and external interfaces, highlighting the continued need for human oversight.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.