Reproduce “Advancing vision-language models in front-end development via data synthesis” on OpenResearch

About this paper

Tong Ge, Yashu Liu, Jieping Ye, Tianyi Li, Chao Wang

Researchers from KE Holdings Inc. and Alibaba Group developed a self-reflective data synthesis pipeline to generate large-scale, high-quality multimodal training data for front-end development, and introduced Flame, a Vision-Language Model for React code generation. The Flame models, trained on this synthetic data, achieved a pass@1 score of up to 52.6% on a new React benchmark, significantly outperforming existing state-of-the-art models including GPT-4o (4.9%).

Key insights

• Explicitly training Vision-Language Models to interpret image layouts as an intermediate step ("Interpretation Before Coding" or 3-stage training) significantly enhances subsequent code generation performance.
• Structured data synthesis methods like Waterfall-Model-Based and Additive Development are more effective than purely evolution-based approaches for generating high-quality, coherent training data for complex front-end code.
• Carefully designed benchmarks, such as `Flame-React-Eval`, are crucial for evaluating the practical utility of generated front-end code, requiring checks for compilation, rendering, and visual similarity.

Summary by alphaXiv.