Reproduce “Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study” on OpenResearch

About this paper

Farahnaz Akrami, Mohammed Samiul Saeef, Qingheng Zhang, Wei Hu, Chengkai Li

This paper demonstrates that prevalent data redundancies in standard Knowledge Graph Completion (KGC) benchmark datasets, such as reverse triples and duplicate relations, artificially inflate the reported performance of embedding models. Re-evaluating models on cleaned datasets reveals a substantial drop in performance and a narrowed gap between complex and simpler models, suggesting that current KGC models may be less effective in real-world scenarios than previously indicated.

Key insights

• Widely used KGC benchmark datasets (FB15k, WN18, YAGO3-10) contain various data redundancies, including reverse triples, duplicate relations, and Cartesian product relations.
• These redundancies artificially inflate the reported performance of embedding-based KGC models, as models can exploit these patterns without learning deeper semantic relationships.
• More complex embedding models show a diminished performance advantage over simpler models or even rule-based systems when evaluated on cleaned datasets, suggesting their reported gains often stem from exploiting data artifacts.

Summary by alphaXiv.