Reproduce “AugmenTest: Enhancing Tests with LLM-Driven Oracles” on OpenResearch

About this paper

Shaker Mahmud Khandaker, Fitsum Kifetew, Davide Prandi, Angelo Susi

AugmenTest, developed by researchers at Fondazione Bruno Kessler, enables large language models to infer correct test oracles from software documentation and developer comments, addressing the "test oracle problem." The framework, particularly with GPT-4o using an extended prompt, achieved a 45.5% True Positive rate at 60% consistency, outperforming the TOGA baseline which reached 8.2%, and operates with a model-agnostic design.

Key insights

• Large Language Models, particularly powerful general-purpose ones like GPT-4o, can effectively infer correct test oracles from natural language documentation and comments.
• Providing extended natural language context, such as details for all methods in a class, generally improves the accuracy of LLM-generated assertions.
• Retrieval-Augmented Generation (RAG) approaches, in their current implementation within AugmenTest, did not consistently outperform simpler prompt variants, suggesting challenges in effectively integrating retrieved structured data for assertion inference.

Summary by alphaXiv.