Reproduce “Cost-Aware Retrieval-Augmentation Reasoning Models with Adaptive Retrieval Depth” on OpenResearch

About this paper

Helia Hashemi, Victor Rühle, Saravan Rajmohan

Microsoft researchers introduce cost-aware retrieval-augmented reasoning models that employ adaptive retrieval depth, enabling large language models to dynamically request more information from search results. This approach achieves an average of 5% higher Exact Match scores and reduces end-to-end latency by 16-20% compared to previous methods, by accounting for the significantly higher cost of token generation over token encoding.

Key insights

• LLM token generation is empirically over 600% slower than encoding retrieved tokens, highlighting the critical need for cost functions that distinguish between these operations.
• Adaptive retrieval depth alone, without explicit cost awareness, can paradoxically increase costs due to aggressive information requests.
• Explicitly optimizing for latency, by penalizing generated tokens more heavily, leads to substantial real-world efficiency gains without sacrificing accuracy.

Summary by alphaXiv.