Reproduce “Stalled, Biased, and Confused: Uncovering Reasoning Failures in LLMs for Cloud-Based Root Cause Analysis” on OpenResearch

About this paper

Evelien Riddell, James Riddell, Gengyi Sun, Michał Antkiewicz, Krzysztof Czarnecki

An empirical study evaluates open-source large language models for cloud-based root cause analysis, revealing low diagnostic accuracy and widespread reasoning failures even in simplified settings. The work also identified that agentic workflows often degrade performance, while specific reasoning failures are strong predictors of incorrect outcomes.

Key insights

• Even in controlled settings, current open-source LLMs demonstrate generally low accuracy and widespread reasoning failures when performing cloud-based RCA.
• Agentic workflows (ReAct, Plan-and-Execute) frequently yielded diminishing or negative returns for overall diagnostic accuracy for most models, despite sometimes improving propagation path validity for more capable LLMs.
• Specific reasoning failures, such as anchoring bias, repetition, arbitrary evidence selection, and failure to update belief, are strong negative predictors of RCA correctness.
• Different alert modalities have distinct impacts: metrics are crucial for localizing root causes, logs aid in classifying fault types, while trace data can sometimes act as a distractor.

Summary by alphaXiv.