Reproduce “Teaching an Agent to Sketch One Part at a Time” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaodan Du, Ruize Xu, David Yunis, Yael Vinker, Greg Shakhnarovich
Researchers developed a method for generating vector sketches one semantic part at a time, guided by natural language text. Their approach combines an automated part annotation pipeline for dataset creation with a two-stage supervised fine-tuning and reinforcement learning framework, achieving higher quality sketches and better human preference compared to previous methods, matching ground truth performance on Long-CLIP similarity.
Key insights
- Automating the annotation of vector sketches with semantic part information via a VLM-driven pipeline is effective and scalable for creating structured datasets.
- Generating vector sketches part-by-part, guided by text, significantly enhances control, interpretability, and localized editability, aligning better with human creative workflows.
- Incorporating dense, intermediate visual rewards in a multi-turn reinforcement learning setup (process-reward GRPO) is crucial for overcoming the SFT-inference distribution gap and improving the quality of autoregressively generated complex visual outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.