Reproduce “Specification Before Generation: A Pre-Registered, Five-Model Paired Evaluation of a Specification Frame for LLM-Generated Code in Money, Time, Idempotency, and Access Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sandeep Dhuri
The authors test whether adding a fixed 267-word domain specification to prompts reduces defects in generated Python backend code. They compare framed and bare prompts across 50 tasks and five model lineages, scoring nine named defect classes. All five models show fewer findings with the frame; pooled counts are 23 versus 148. The study does not measure correctness or isolate the frame’s content from added text.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.