Reproduce “AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tiancheng Huang, Ruisheng Cao, Yuxin Zhang, Zhangyi Kang, Zijian Wang, Chenrun Wang, Yijie Luo, Hang Zheng and 3 more
The X-LANCE Lab introduces AirQA, a comprehensive, multi-task, multi-modal QA dataset with instance-level evaluation, alongside EXTRACTOR, an automated multi-agent framework for synthesizing high-quality interaction trajectories for LLM agents. These tools establish a realistic benchmark for AI research QA and enable automated generation of training data for interactive scientific QA agents.
Key insights
- State-of-the-art LLMs, both proprietary and open-source, underperform on the AirQA benchmark, indicating current limitations in handling complex, multi-modal scientific information and long-term planning for scientific QA tasks.
- Automated instruction data synthesis via the EXTRACTOR framework effectively generates high-quality interaction trajectories, enabling smaller LLM agents to significantly improve multi-turn tool-use capabilities and achieve performance comparable to larger, untrained models.
- Instance-level evaluation using objective, function-based metrics provides a more precise and reliable assessment of factual correctness for scientific QA compared to traditional linguistic metrics or subjective LLM-as-a-judge approaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.