Reproduce “A Network Arena for Benchmarking AI Agents on Network Troubleshooting” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhihao Wang, Alessandro Cornacchia, Alessio Sacco, Franco Galante, Marco Canini, Dingde Jiang

NIKA introduces the first large-scale public benchmark and orchestration framework for evaluating AI agents in dynamic network troubleshooting scenarios. Its evaluation of state-of-the-art LLMs demonstrates that GPT-5 achieved 89.0% detection accuracy but only 55.3% root cause analysis accuracy, highlighting significant challenges for LLMs in complex network diagnosis.

Key insights

  • The structured Agent Access Layer (AAL) is effective in reducing erroneous tool invocations by LLMs, with GPT-5 showing only a 0.7% error rate, suggesting robust tool integration.
  • While larger LLMs like GPT-5 detect network issues well, their performance in precise fault localization and root cause analysis significantly degrades with increasing network size and complexity.
  • Current LLMs exhibit 'shallow reasoning' for complex issues like resource contention, often failing to connect direct symptoms to deeper underlying causes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.