Reproduce “FiTs: Fine-grained Two-stage Training for Knowledge-aware Question Answering” on OpenResearch

About this paper

Qichen Ye, Bowen Cao, Nuo Chen, Weiyuan Xu, Yuexian Zou

Researchers from Peking University and Peng Cheng Laboratory developed FiTs, a two-stage training framework for Knowledge-aware Question Answering (KAQA) that bridges semantic and distributional gaps between language models and knowledge graphs. The framework consistently achieved state-of-the-art results across three benchmarks, including a 2.6% absolute accuracy improvement on CommonsenseQA, with only 1% additional trainable parameters.

Key insights

• Explicitly aligning representations from PLMs and KGs through a post-training stage (KA loss) is crucial for effectively bridging modality gaps and provides a superior initialization for subsequent fine-tuning.
• Self-supervised learning objectives like Knowledge Source Distinction (KSD) effectively train models to discriminate relevant knowledge from noise in retrieved KG sub-graphs.
• Knowledge Backbone Regularization (KBR), which reinforces KG structure, is most effective when combined with KSD, suggesting that understanding knowledge relevance is a prerequisite for robust structural reasoning.

Summary by alphaXiv.