Reproduce “DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queries” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianyou Wang, Kaicheng Wang, Xiaoyue Wang, Prudhviraj Naidu, Leon Bergen, Ramamohan Paturi
DORIS-MAE introduces a benchmark for retrieving scientific papers from long, multifaceted research questions rather than short single-intent queries. Using hierarchical aspect annotations and validated LLM-assisted labeling, it shows that current retrieval models struggle with complex scientific information needs, while supervised adaptation can raise E5-Large-v2’s MAP from 40.62 to 49.15.
Key insights
- Retrieval becomes substantially harder as queries combine more heterogeneous requirements: two-aspect sub-queries consistently outperform full-query retrieval, while composite interdisciplinary queries are especially difficult.
- Explicit aspect information can improve some models, but simply concatenating aspects does not reliably help, indicating a need for retrieval architectures that model interactions among requirements.
- ChatGPT–expert agreement falls within the observed range of human–human disagreement on the validation set, supporting LLM-assisted annotation when prompt development, held-out testing, and expert checks are used.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.