Reproduce “WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingde Xu, Zhen Yang, Wenyi Hong, Lihang Pan, Xinyue Fan, Yan Wang, Xiaotao Gu, Bin Xu and 1 more
Researchers from Meituan, Zhejiang University, and University of Waterloo developed WebVIA, an agentic framework that generates interactive and verifiable web UI-to-Code. This framework employs an exploration agent to discover UI states and a UI2Code model to synthesize functional HTML/CSS/JavaScript, achieving an 85.30% F1 score for action generation and an 84.9% task completion ratio on new interactive benchmarks.
Key insights
- An agentic framework that integrates systematic UI exploration, interactive code generation, and functional validation is essential to bridge the gap between static UI design and dynamic web development.
- Specialized models, fine-tuned on multimodal datasets encompassing multi-state UI observations and explicit interaction traces, outperform general-purpose vision-language models for both UI exploration and interactive code synthesis.
- Developing new, flow-based benchmarks that emphasize behavioral ground truth and task completion provides a more rigorous and relevant evaluation standard for interactive UI-to-Code generation than traditional visual fidelity metrics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.