Skip to search boxSkip to navigationSkip to main content

Enhancing text comprehension via fusing pre-trained language model with knowledge graph

  • Jing Qian
    ,
  • Gangmin Li
    ,
  • Katie Atkinson
    ,
  • Yong Yue
  • Xi'an Jiaotong-Liverpool University
    ,
  • University of Liverpool
Research Output: Chapter in Book/Report/Conference proceeding Conference contribution Peer-review

Abstract

Pre-trained language models (PLMs) such as BERT and GPTs capture rich linguistic and syntactic knowledge from pre-training over large-scale text corpora, which can be further fine-tuned for specific downstream tasks. However, these models still have limitations as they rely on knowledge gained from plain text and ignore structured knowledge such as knowledge graphs (KGs). Recently, there has been a growing trend of explicitly integrating KGs into PLMs to improve their performance. For instance, K-BERT incorporates KG triples as domain-specific supplements into input sentences. Nevertheless, we have observed that such methods do not consider the semantic relevance between the introduced knowledge and the original input sentence, leading to the issue of knowledge impurities. To address this issue, we propose a semantic matching-based approach that enriches the input text with knowledge extracted from an external KG. The architecture of our model comprises three components: the knowledge retriever (KR), the knowledge injector (KI), and the knowledge aggregator (KA). The KR, built upon the sentence representation learning model (i.e. CoSENT), retrieves triples with high semantic relevance to the input sentence from an external KG to alleviate the issue of knowledge impurities. The KI then integrates the retrieved triples from the KR into the input text by converting the original sentence into a knowledge tree with multiple branches, the knowledge tree is transformed into an accessible sequence of text that can be fed into the KA. Finally, the KA takes the flattened knowledge tree and passes it through an embedding layer and a masked Transformer encoder. We conducted extensive evaluations on eight datasets covering five text comprehension tasks, and the experimental results demonstrate that our approach exhibits competitive advantages over popular knowledge-enhanced PLMs such as K-BERT and ERNIE.

Publication Information

Output type

Research Output: Chapter in Book/Report/Conference proceeding Conference contribution Peer-review

Host publication Subtitle

2023 6th International Conference on Algorithms, Computing and Artificial Intelligence

Original language

English

Pages from-to (Number of pages)

Pages 353-360 (8 pages)

Publication milestones

  • Published - 16/02/2024

Publication status

Published - 16/02/2024

Publisher

Association for Computing Machinery, United States

Publication series

  • Publication series name: ACM International Conference Proceeding Series

ISBN (Print)

9798400709203

ISBN (Electronic)

9798400709203

External Publication IDs

  • handle.net: 10547/626198
  • Scopus: 85185825645

Host publication title

ACAI '23: Proceedings of the 2023 6th International Conference on Algorithms, Computing and Artificial Intelligence

Access to documents

Publication metrics