Reproduce “Synthesizing Performance Constraints for Evaluating and Improving Code Efficiency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jun Yang, Cheng-Chi Wang, Bogdan Alexandru Stoica, Kexin Pei
WEDGE, a framework developed by researchers at The University of Chicago, combines Large Language Models (LLMs) with coverage-guided fuzzing to create performance-stressing tests for code. It generates tests that achieve an average slowdown of 363x over default tests, leading to a 24% reduction in CPU instructions and uncovering previously missed optimization opportunities in LLM-optimized code.
Key insights
- Decomposing the complex task of generating performance-stressing tests into LLM-driven constraint synthesis and fuzzer-guided input exploration effectively combines the strengths of both semantic understanding and scalable search.
- Providing LLMs with detailed contrastive execution feedback (line-level hit counts) significantly enhances their ability to infer and formulate precise, natural language constraints that characterize performance bottlenecks.
- Using LLM-derived performance constraints as explicit coverage targets for fuzzing allows for efficient discovery of inputs that trigger subtle, implementation-specific inefficiencies, rather than just large input sizes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.