Reproduce “A Benchmark Construction and Evaluation Framework for Specialist Domains: Case Study on Defense-related Documents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bao Gia Doan, Aditya Joshi, Pantelis Elinas, Aarya Bodhankar, Oscar Leslie, Tom Marchant, Flora Salim
DoRA creates an in-domain benchmark and training data from specialist documents, linking generated questions and answers to supporting passages. It evaluates answers for grounding, then adapts Llama-3.1-8B using LoRA fine-tuning. In a defense-document case study, adaptation lowered the RAGEval hallucination score from 4.44% to 2.17% under GTE top-3 retrieval; this is a keypoint measure, not evidence of cross-domain transfer.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.