Reproduce “Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hui Yang, Sifu Yue, Yunzhong He
An evaluation of Auto-GPT styled agents on online decision-making tasks like WebShop and ALFWorld demonstrates GPT-4's superior performance compared to other large language models. The study also introduces "Additional Opinions," an algorithm that integrates expert advice to improve agent performance, boosting GPT-4's success rate on WebShop from 0.240 to 0.320.
Key insights
- GPT-4 consistently exhibits superior performance over GPT-3.5, Claude, and Vicuna when operating as an Auto-GPT styled agent in online decision-making tasks.
- Integrating "Additional Opinions" from expert models, even single suggestions, can substantially improve the performance of LLM agents, especially for models with strong discernment like GPT-4.
- The effectiveness of incorporating external advice is highly dependent on both the quality of the provided opinions and the LLM's capacity to critically evaluate and filter that information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.