Reproduce “DeepEra: A Deep Evidence Reranking Agent for Scientific Retrieval-Augmented Generated Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haotian Chen, Qingqing Long, Siyu Pu, Xiao Luo, Wei Ju, Meng Xiao, Yuanchun Zhou, Jianghua Zhao and 1 more
DeepEra introduces an agentic reranking framework for scientific Retrieval-Augmented Generation (RAG) that leverages Large Language Model (LLM) reasoning to filter "semantically similar but logically irrelevant" (SSLI) evidence. This approach enhances the factual reliability and accuracy of LLM-generated answers, achieving a HitRate@1 of 66.6% and a Logic Fidelity Score of 3.94 in challenging SSLI settings, surpassing leading baselines.
Key insights
- Existing embedding models struggle significantly to filter "semantically similar but logically irrelevant" (SSLI) distractors, confirming a fundamental limitation of current RAG approaches that compromises factual reliability.
- Integrating explicit LLM-powered reasoning directly into the reranking process, as an agentic component, effectively bridges sophisticated reasoning with robust evidence filtering, moving beyond surface-level semantic similarity.
- Specialized datasets, like SciRAG-SSLI, which systematically introduce adversarially generated logical distractors, are essential for rigorously evaluating and improving the logical robustness of RAG systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.