Reproduce “JavelinGuard: Low-Cost Transformer Architectures for LLM Security” on OpenResearch

About this paper

Yash Datta, Sharath Rajasekar

Researchers from Javelin and Columbia University develop JavelinGuard, a suite of five progressively sophisticated low-cost transformer architectures that achieve superior cost-performance trade-offs for LLM security, with their most advanced model Raudra achieving 96.2% accuracy on challenging JavelinBench evaluations while maintaining 38.5ms inference latency (25-40x faster than GPT-4o's 800ms) and demonstrating robust differentiation between jailbreak attacks, prompt injection, and benign content through multi-task learning with attention-weighted pooling and Focal Loss optimization, outperforming existing open-source guardrails like InjecGuard while enabling practical CPU-based deployment for real-time production environments.

Key insights

• High-performance LLM security can be achieved with compact transformer architectures (~400M parameters) without requiring large decoder-only models
• Multi-task learning with separate classification heads for different attack types significantly improves detection granularity and reduces false positives
• Attention-weighted pooling and Focal Loss optimization effectively address class imbalance and hard-to-classify examples in adversarial datasets
• Low inference latency enables practical prompt segmentation strategies to mitigate the "lost in the middle" problem

Summary by alphaXiv.