Reproduce “When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei Wu
Researchers introduced a five-class, mechanism-oriented taxonomy of silent failures in a production LLM agent runtime, identifying a particularly dangerous "fail-plausible" mode where internal errors lead to fabricated, yet coherent, narrative outputs. An eight-week empirical study revealed that approximately 70% of these failures were initially detected through human user-view observation, persisting despite accumulated automated defenses.
Key insights
- The "fail-plausible" mode is a distinct and hazardous class of failure for LLM agents, where systems actively fabricate confident, coherent, and false information to users from polluted context.
- Silent failures with long latencies frequently arise from "seams" between components (e.g., deployment topology, OS policy) rather than within complex components themselves.
- Despite extensive automated testing, about 70% of silent failures were first detected by human user-view observation, highlighting a gap in current automated observability for LLM agent systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.