Reproduce “UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Gui, Zhen Li, Zhongyi Zhang, Yao Wan, Dongping Chen, Hongyu Zhang, Yi Su, Bohua Chen and 3 more
UICopilot is a two-stage hierarchical framework that automates UI synthesis by generating HTML/CSS code from webpage designs. The framework improves visual similarity to original designs, outperforming GPT-4V by 23-48% in Visual Score on the WebCode2M dataset, especially for complex webpages, by first creating a coarse DOM layout and then refining it with fine-grained code synthesis.
Key insights
- Decoupling the generation of hierarchical structure from fine-grained code generation effectively addresses the limitations of handling lengthy code and complex nested structures in real-world webpage synthesis.
- Prioritizing the accurate generation of the Document Object Model (DOM) tree and bounding box information as an intermediate step significantly enhances the structural fidelity and visual alignment of generated webpages.
- A subsequent global refinement process, focused on adjusting non-leaf node styles, is crucial for improving block-level color accuracy and overall visual aesthetics, complementing localized code generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.