Reproduce “VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jie Deng, Kaichun Yao, Libo Zhang

VisRefiner introduces a training framework that enables multimodal large language models (MLLMs) to learn directly from visual differences between generated and target UI designs, leading to significantly enhanced visual fidelity and layout consistency in frontend code generation. This approach teaches MLLMs to self-correct based on perceptual feedback, improving initial generation quality and refinement stability.

Key insights

  • Training MLLMs to directly interpret visual differences and map them to code edits significantly improves both initial generation quality and the ability to self-correct.
  • A composite dataset of synthetically perturbed code and real model errors provides a robust learning signal for visual-difference-driven refinement.
  • Reinforcement learning, particularly with GRPO and a perceptual similarity-based reward, effectively instills stable and consistent self-refinement capabilities in MLLMs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.