Reproduce “Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zheng Luo, T Pranav Kutralingam, Ogochukwu N Okoani, Wanpeng Xu, Hua Wei, Xiyang Hu
This research reveals that Large Language Models frequently encounter "parameter value language mismatch" when performing tool calling with non-English user queries, even when the underlying intent is understood. These language boundary failures lead to a significant increase in execution-level errors, highlighting a critical gap between multilingual semantic understanding and reliable tool interaction.
Key insights
- Multilingual semantic understanding in LLMs does not automatically translate to robust multilingual tool execution, as language-invariant execution interfaces create a significant bottleneck.
- The primary failure mode in multilingual tool calling is "parameter value language mismatch," where semantically correct non-English values violate English-only execution conventions.
- Effective solutions require better alignment between natural language interaction and execution conventions, suggesting that prompt engineering or post-processing alone are insufficient for comprehensive robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.