Reproduce “EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhong Sun, Joachim Rahmfeld, Chris Weaver, Weijia Chen, Roshan Desai, Wenxi Huang, Mark H. Butler
EnterpriseRAG-Bench introduces a synthetic corpus of 500,000 documents and 500 questions, designed to evaluate Retrieval-Augmented Generation (RAG) systems against the unique challenges of company-internal knowledge. Evaluation with baseline RAG systems demonstrated that vector search underperforms keyword search and iterative agents on this domain, suggesting existing models struggle with enterprise-specific data characteristics.
Key insights
- Enterprise data exhibits higher local density and global clustering compared to open-domain corpora, implying greater challenge for retrieval due to numerous semantically similar distractor documents.
- Existing RAG models and embeddings trained on public data may have limited efficacy with enterprise-specific vocabulary, structured formats, and conversational styles, as shown by vector search's underperformance.
- A dynamic, correction-aware evaluation framework with multi-retriever pooling and human curation is essential for building a robust benchmark, allowing ground truth to improve over time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.