Reproduce “CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, Heng Ji

The CREATOR framework empowers Large Language Models (LLMs) to dynamically create and utilize their own custom tools, moving beyond reliance on pre-defined APIs. This approach, developed by researchers at Tsinghua University and the University of Illinois at Urbana-Champaign, disentangles abstract and concrete reasoning, incorporating a robust self-correction mechanism. CREATOR demonstrates superior performance on complex reasoning benchmarks like MATH and TabMWP, achieving 59.7% and 94.7% accuracy respectively, while also proving effective in solving problems that explicitly require novel tool generation.

Key insights

  • Disentangling abstract reasoning (tool creation) from concrete reasoning (tool application) consistently improves LLM performance.
  • A rectification stage that utilizes execution tracebacks enables LLMs to effectively self-correct errors in generated code, improving accuracy by approximately 10%.
  • LLMs demonstrate varying levels of intrinsic tool creation, from enhancing existing tools to creating hierarchical structures, and can transfer created tools across conceptually similar problems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.