Reproduce “CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiayi Li, Ziyuan Wang, Daniel Garijo, María Poveda-Villalón

CQ4OE evaluates language models’ ontology generation for natural-language competency questions. It links each question to required terms and TBox axioms, then scores vocabulary and structure both per question and overall. Across nine models, CQ2Onto class recovery averaged 59.7% F1, versus 31.8% for properties and 16.7% for hierarchy closure; this is a benchmark evaluation, not a comparison with other systems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.