Reproduce “RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyuan Peng, Xin Yin, Pu Zhao, Fangkai Yang, Lu Wang, Ran Jia, Xu Chen, Qingwei Lin and 2 more
RepoGenesis introduces the first multilingual benchmark for evaluating end-to-end web microservice generation, assessing LLMs and coding agents on their ability to create complete, deployable repositories from natural language requirements. It reveals that current commercial IDEs achieve higher functional correctness (e.g., Copilot at 23.67% Pass@1 for Python) and deployability compared to open-source agents, with a notable 'API Coverage - Deployment Success Rate' gap.
Key insights
- A significant performance disparity exists between commercial IDEs and open-source agents, with commercial tools consistently outperforming open-source counterparts in microservice generation.
- The identified "API Coverage - Deployment Success Rate" (AC-DSR) gap indicates that models can often generate structural code (API skeletons) but struggle with the precision required for successful system deployment and initialization.
- Performance degrades substantially as repository complexity increases, particularly for smaller models, highlighting persistent challenges in architectural reasoning and handling cross-file consistency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.