Reproduce “Tiny-Critic RAG: Empowering Agentic Fallback with Parameter-Efficient Small Language Models” on OpenResearch

About this paper

Yichao Wu, Penghao Liang, Yafei Xiang, Mengwei Yuan, Jianan Liu, Jing Yang, Xianyou Li, Weiran Yan

Tiny-Critic RAG introduces a system that employs a parameter-efficient Small Language Model (SLM) to preemptively evaluate retrieved context in agentic RAG systems, achieving a 94.6% reduction in routing latency and a 98% decrease in operational costs compared to large LLM-based evaluators, while maintaining high factual accuracy.

Key insights

• Parameter-efficient SLMs, when specifically fine-tuned with methods like LoRA, can perform highly accurate and deterministic binary classification tasks comparable to larger LLMs for critical system routing.
• Optimized inference techniques like constrained decoding and non-thinking mode can drastically reduce the latency of SLM-based evaluation, making reflective RAG practically feasible for real-time applications.
• Preemptive evaluation of retrieved context prevents costly multi-hop reasoning spirals and redundant actions by autonomous agents, leading to significant implicit cost savings in addition to explicit cost reductions.

Summary by alphaXiv.