Reproduce “Towards Knowledgeable Deep Research: Framework and Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenxuan Liu, Zixuan Li, Long Bai, Chunmao Zhang, Fenghui Zhang, Zhuo Chen, Wei Li, Yuxin Zuo and 8 more
Researchers from the Chinese Academy of Sciences introduced the Knowledgeable Deep Research (KDR) task and the Hybrid Knowledge Analysis (HKA) framework, enabling large language model agents to generate comprehensive, multimodal reports by reasoning over both structured and unstructured knowledge. The accompanying KDR-Bench benchmark and evaluation framework demonstrated HKA's ability to achieve higher key point coverage (61.7%) and supportiveness (27.8%) on structured data, and superior visual report quality, outperforming most baselines.
Key insights
- Explicitly processing structured knowledge through dedicated components, like the S.K.A.'s use of code models for computation and vision-language models for insight generation, is crucial for deep analysis and deriving novel conclusions.
- The combination of structured and unstructured knowledge analysis is essential, as evidenced by significant performance drops when either component is removed, validating the hybrid approach for comprehensive report generation.
- Evaluating multimodal reports with vision-enhanced metrics reveals capabilities missed by traditional text-only assessments, highlighting the importance of visual features and presentation quality in research documentation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.