Reproduce “Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianyi Xiong, Zhengyuan Yang, Xiaofei Wang, Chung-Ching Lin, Ruichun Ma, Kevin Lin, Zhendong Wang, Linjie Li and 5 more
RubSE, a framework for UI-to-code generation, leverages single-step visual rubrics to guide iterative code refinement, addressing the issue of "visual repair coupling." This approach consistently improves final-round and best-round performance across various large vision-language models, demonstrating enhanced stability and more effective long-tail error correction compared to naive self-evolution.
Key insights
- Representing visual feedback as structured rubrics, especially abstract and action-oriented ones, effectively mitigates visual repair coupling by focusing the model's attention on specific discrepancies.
- Maintaining a cross-round history of applied rubrics prevents redundant repairs and guides the model towards addressing novel, long-tail visual mismatches, leading to more consistent improvements.
- High-quality rubrics generated by stronger VLMs can act as model-agnostic visual-repair context, effectively transferring guidance to improve the self-evolution of weaker code generators.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.