Reproduce “J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chihiro Yano, Jun Hirako, Hiroaki Yamagiwa, Ryota Hirobuchi
J-tau localizes tau-bench for Japanese customer-support evaluation across airline, retail, and telecom. It translates policies, tools, data, and instructions, adapts entities, and uses expert review, task repair, and scoring changes counting agent-side output errors as failures. For Qwen3.5-4B, telecom scoring changes from 98.7 to 52.0, showing original aggregation can inflate scores. J-tau covers three domains, not every Japanese service setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.