Reproduce “On the Robustness of Agentic Function Calling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ella Rabinovich, Ateret Anaby-Tavor
IBM Research investigated the robustness of large language model agents' function calling capabilities under naturalistic user query variations and expanded toolkits. The study found consistent performance drops across models and identified that a large share of failures with rephrased queries were artifacts of exact-match evaluation rather than agent misbehavior.
Key insights
- State-of-the-art LLM agents exhibit consistent performance degradation when faced with naturalistic query variations or expanded toolkits, indicating a lack of robustness despite high baseline accuracy.
- A significant portion of apparent failures due to rephrased queries is an artifact of overly strict exact-match evaluation in current benchmarks, overestimating actual agent vulnerability.
- Expanding toolkits with semantically related functions leads to diverse agent misbehaviors, primarily wrong function selection and incorrect parameter assignment, highlighting challenges in disambiguation and argument grounding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.