Reproduce “EmbC-Test: How to Speed Up Embedded Software Testing Using LLMs and RAG” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maximilian Harnot, Sebastian Komarnicki, Michal Polok, Timo Oksanen
EmbC-Test combines hybrid retrieval-augmented generation with large language models to draft executable tests for embedded C software using project-specific source code, requirements, documentation, and legacy tests. In an industrial evaluation, the workflow reached 100% syntactic validity, 84.5% runtime validity, about 270 generated tests per hour, and an estimated 66% reduction in testing effort, while still requiring expert review for assertions and edge cases.
Key insights
- Retrieval grounding improves practical compatibility with an embedded software environment: runtime validation reached 84.5% with RAG, compared with 62.4% for random retrieval and 50.5% without retrieval.
- Hybrid retrieval is well suited to source code because semantic search finds related concepts while lexical search preserves exact matches for functions, types, macros, and APIs.
- Generated tests are effective as first drafts but do not replace expert testing: their edge-case completeness was weaker than other human ratings, and mature manual suites still achieved higher coverage.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.