Reproduce “PaT: Planning-after-Trial for Efficient Test-Time Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Youngsik Yoon, Sungjae Lee, Seockbean Song, Siwei Wang, Wei Chen, Jungseul Ok
An adaptive strategy named Planning-after-Trial (PaT) was developed by researchers at POSTECH and Microsoft Research Asia to enhance the efficiency of LLM-based code generation. This approach defers costly planning until an initial direct solution attempt fails verification, improving both performance and cost-efficiency by achieving higher Pass@1 while reducing inference costs by up to 60% compared to prior rigid planning methods.
Key insights
- Deferring complex planning until an initial solution attempt has definitively failed significantly reduces unnecessary computational overhead compared to static planning approaches.
- An adaptive, failure-triggered intervention policy, guided by execution feedback, is more cost-effective and achieves better performance for varied problem complexities.
- Optimizing resource allocation through a heterogeneous model setup, where a smaller model generates initial attempts and a larger model plans only for difficult cases, improves the overall cost-performance trade-off.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.