Reproduce “ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinking” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yunzhe Li, Jianan Wang, Hongzi Zhu, James Lin, Shan Chang, Minyi Guo

Researchers at Shanghai Jiao Tong University developed "ThinkTrap," a black-box denial-of-service (DoS) framework that generates adversarial prompts to compel Large Language Models (LLMs) into extensive, resource-exhausting generations. The attack caused a 100x increase in Time to First Token and reduced throughput to 1% on a self-hosted LLM service, with prompt generation costing mere cents.

Key insights

  • LLMs exhibit a widespread vulnerability to "infinite thinking" induced by specific adversarial prompts, exploitable through black-box access with low attack costs.
  • The Supervised Fine-Tuning (SFT) phase of LLMs is a critical factor, as attack prompts show strong transferability among models fine-tuned on the same SFT dataset.
  • Common defenses like output length limits, basic rate limiting, and n-gram-based anomaly detection are largely insufficient or severely degrade legitimate user experience.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.