Reproduce “Closed-loop evaluation of LLM agents for embedded software development” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté, Juan Trujillo
The authors introduce a benchmark for whether firmware agents can verify and repair controller behavior, rather than merely write plausible code. Agents receive a prose handoff and use a constrained workspace to edit and test firmware in simulation before hidden evaluation. Hosted gpt-5.4 passes 14/15 runs (93.3%); qwen3.5-27B passes 9/15 (60.0%). Results are limited to a small, simulated control suite.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.