Reproduce “Learning API Functionality from In-Context Demonstrations for Tool-based Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bhrij Patel, Ashish Jagmohan, Aditya Vempaty
Researchers introduced and formalized the problem of enabling tool-based LLM agents to learn API functionality solely from in-context demonstrations, bypassing the need for explicit documentation. They empirically demonstrated that current state-of-the-art LLMs struggle significantly with accurate parameter inference and identified that natural language guidelines derived from agent self-exploration can improve task success by 12% in some environments.
Key insights
- Accurate parameter information, including schema, types, constraints, and formatting, is the most critical and challenging aspect for LLMs to infer solely from demonstrations.
- Misleading or uninformative API error messages significantly hamper an agent's ability to learn and self-correct, preventing effective refinement of functionality understanding.
- Simply increasing the quantity of demonstrations (N) does not consistently improve performance; it can sometimes lead to degradation due to context length issues or misinterpretation by the LLM.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.