Reproduce “From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee
Researchers from the University of Wisconsin–Madison and Adobe Research developed an experiential learning framework for open-ended, long-horizon image editing. This framework decomposes abstract natural language instructions into sub-tasks and orchestrates diverse editing tools, demonstrating superior instruction adherence and visual coherence on various editing benchmarks.
Key insights
- Decomposing complex, abstract image editing into distinct planning and orchestration stages, coupled with outcome-based MLLM rewards, creates a flexible and robust system for multi-step visual modifications.
- Self-supervised plan generation guided by a checklist improves the planner's coherence and task-awareness, aligning generated plans with the model's capabilities and reducing distribution shift.
- Efficient reward approximation techniques allow for tractable training of the orchestrator despite the high computational cost of diffusion-based editing, enabling learning from empirical outcomes.
- A closed-loop plan refinement process, where infeasible sub-tasks are pruned, ensures better coordination between the high-level plan and the executable editing tools.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.