Reproduce “LiveEvalBench: Toward Open-World Evaluation for Web Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiyao Wang, Zhen Wen, Yinghao Tang, Yixiao Fu, Lin Yuan, Xiaolau Zhang, Jun Zhou, Wei Chen
LiveEvalBench introduces an agentic, adaptive, and extensible automated framework for "open-world" evaluation of large language model-generated web projects. The framework provides fine-grained diagnostic insights into model performance, effectively differentiating models and demonstrating strong alignment with human judgments, particularly highlighting runtime UI interaction as a critical challenge.
Key insights
- Runtime UI interaction capabilities present the most significant challenge and differentiator for current large language models in web generation tasks.
- The multi-agent, role-separated evaluation design offers finer diagnostic granularity, effectively capturing a broader spectrum of performance nuances compared to single-agent methods.
- The framework's automated UI assessments demonstrate strong quantitative agreement with human expert judgments, affirming its reliability in capturing qualitative aspects of user experience.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.