Reproduce “Exploring the Latest LLMs for Leaderboard Extraction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Salomon Kabongo, Jennifer D'Souza, Sören Auer
This study evaluates open-source (Llama-2 7B, Mistral 7B) and proprietary (GPT-4-Turbo, GPT-4.o) Large Language Models for extracting (Task, Dataset, Metric, Score) quadruples from AI papers, demonstrating that fine-tuned Mistral 7B achieves competitive or superior performance, particularly with tailored contextual inputs like DocTAET and DocREC.
Key insights
- Fine-tuned open-source LLMs, especially Mistral 7B, can achieve competitive or superior performance compared to proprietary models in domain-specific scientific information extraction tasks.
- Carefully selected, tailored contexts (DocTAET and DocREC) consistently outperform using the full document (DocFULL), which often leads to reduced accuracy and distraction for LLMs.
- The DocREC context on Mistral 7B provided a balanced trade-off between precision and other performance metrics for extracting leaderboard information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.