Reproduce “APE: Agentic Prompt Enhancer for Image Generation and Editing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zijian Huang, Jay Zhangjie Wu, Zian Wang, Tianshi Cao, Jiasi Chen, Sanja Fidler, Huan Ling, Xuanchi Ren
NVIDIA researchers developed APE, an agentic framework that post-trains Small Language Models (SLMs) to enhance user prompts for image generation and editing. This approach, utilizing multi-agent designs and reinforcement learning, achieves performance comparable to large proprietary LLMs, narrowing the performance gap from -9% to between -3% and +2% on various benchmarks while improving visual alignment without altering the visual backbone.
Key insights
- Reinforcement learning, directly optimizing against visual outputs, effectively trains SLMs for prompt enhancement, significantly improving visual alignment and prompt following.
- Multi-agent architectures (MAPE) can decompose complex prompt enhancement tasks into specialized roles, leading to improved grounding of fine-grained details and better handling of compositional reasoning.
- Small Language Models (SLMs), when post-trained with task-aware RL and structured agentic designs, can achieve prompt enhancement performance on par with powerful, closed-source LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.