Reproduce “Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yutao Sun, Yanting Miao, Hao-Xuan Ma, Mengyu Zhou, Mingshuai Chen, Tiancheng Zhao, Dexin Wang, Lei Lv and 3 more
Dynamo adapts frozen vision-language models by evolving reusable reasoning skills and executable image-processing tools from a small labeled subset. An analyzer diagnoses whether failures are cognitive or perceptual, and a loop promotes capabilities that improve training accuracy. Across five backbones and four benchmarks, the system gains 5.6 percentage points on average in 20 settings, while requiring supervision and API calls.
Key insights
- Frozen VLMs can autonomously identify and address their own performance bottlenecks by generating complementary reasoning skills and visual tools without requiring gradient updates.
- Visual diagnosis (inspecting artifacts and errors visually) is crucial for generating effective image-processing tools, as text-only analysis often fails to produce usable visual transformations.
- Attaching 'mastery skills' to tools, specifying when to use them and their potential pitfalls, enables selective and beneficial tool invocation, preventing performance degradation from indiscriminate application.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.