Reproduce “Boosting LLMs for Mutation Generation” on OpenResearch

About this paper

Bo Wang, Ming Deng, Mingda Chen, Chengran Yang, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang

Researchers from Beijing Jiaotong University and collaborators developed SMART, a framework leveraging retrieval-augmented generation, logic-based code chunking, and supervised fine-tuning to improve the quality of mutations generated by Large Language Models for software testing. The approach yields more valid and effective mutations, enhancing test case prioritization and fault localization, and allows 7B-scale open-source models to compete with larger commercial LLMs.

Key insights

• Combining logic-based code chunking, RAG, and SFT systematically enhances mutation validity (generation, non-duplicate, compilable rates) and effectiveness (real-bug detection, semantic similarity).
• Logic-based code chunking contributes most significantly to individual performance improvements, while fine-tuning is crucial for adapting mid-scale LLMs to the mutation generation task.
• SMART enables 7B-scale open-source LLMs to achieve performance competitive with, and in some cases superior to, larger proprietary models like GPT-4o.

Summary by alphaXiv.