Reproduce “KG-MuLQA: A Framework for KG-based Multi-Level QA Extraction and Long-Context LLM Evaluation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nikita Tatarinov, Vidhyakshaya Kannan, Haricharana Srinivasa, Arnav Raj, Harpreet Singh Anand, Varun Singh, Aditya Luthra, Ravij Lade and 2 more

The KG-MuLQA framework, developed by researchers at Georgia Institute of Technology, enables fine-grained, multi-level assessment of long-context large language models by programmatically extracting complex question-answer pairs from financial documents using knowledge graphs. Evaluations on 16 LLMs reveal a consistent decline in performance as question complexity increases, highlighting systematic failures in multi-hop reasoning and set operations.

Key insights

  • Representing document information as knowledge graphs allows for systematic, scalable, and precise extraction of QA pairs while preserving real document semantics.
  • Defining question complexity across orthogonal dimensions (multi-hop retrieval, set operations, answer plurality) enables a nuanced and interpretable analysis of LLM performance on controlled reasoning tasks.
  • Programmatic QA generation from KGs provides a robust and contamination-resistant method for creating benchmarks that rigorously test complex reasoning over long, domain-specific texts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.