Reproduce “Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts” on OpenResearch

About this paper

Haoyu Dong

Visual-SDPO, a self-distillation policy optimization framework from Microsoft, integrates rendered visual feedback and a region-to-code mapping into large language model training to address the 'code-vision gap' in generating visual artifacts. The method improves the visual quality of charts, web pages, and presentation slides by over 10 absolute points compared to zero-shot baselines, and is more sample-efficient than visual-reward RL methods, all without adding inference-time overhead.

Key insights

• Integrating rendered visual feedback as privileged context within a self-distillation framework effectively bridges the 'code-vision gap' in LLM training, allowing models to learn from visual outcomes without inference-time overhead.
• Spatially targeted distillation gradients, achieved by linking visual defects to specific code statements, significantly improve learning efficiency and precision compared to uniform token-level or sparse trajectory-level feedback.
• The combination of dense, localized token-level visual feedback with a sequence-level execution-grounded reward provides robust and sample-efficient training for generating high-quality visual artifacts across various domains.

Summary by alphaXiv.