Reproduce “ControlLLM: Augment Language Models with Tools by Searching on Graphs” on OpenResearch

About this paper

Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Ziheng Li, Xizhou Zhu, Lewei Lu, Qifeng Chen and 3 more

ControlLLM, developed by researchers from OpenGVLab and The Hong Kong University of Science and Technology, enhances large language models' ability to use multi-modal tools by employing a graph-based search paradigm called Thoughts-on-Graph. This framework significantly improves tool selection, argument assignment, and overall task resolution for complex multi-modal problems, achieving a 0.93 solution evaluation score and outperforming baselines on difficult tasks.

Key insights

• Explicitly modeling tool dependencies and resource flow within a graph structure, rather than relying solely on LLM-generated sequential or tree-like plans, leads to superior accuracy and robustness in tool orchestration.
• An adaptive search strategy, balancing exploration and exploitation on the tool graph, provides a practical trade-off between solution quality and computational efficiency, making it suitable for real-world deployment.
• Combining instruction-tuned smaller LLMs for task decomposition with powerful zero-shot LLMs for complex reasoning modules (like tool assessment and resource inference) offers a robust and effective hybrid approach.

Summary by alphaXiv.