JKRL: Joint Knowledge Representation Learning of Text Description and Knowledge Graph

Abstract

The purpose of knowledge representation learning is to learn the vector representation of research objects projected by a matrix in low-dimensional vector space and explore the relationship between embedded objects in low-dimensional space. However, most methods only consider the triple structure in the knowledge graph and ignore the additional information related to the triple, especially the text description information. In this paper, we propose a knowledge graph representation model with a symmetric architecture called Joint Knowledge Representation Learning of Text Description and Knowledge Graph (JKRL), which models the entity description and relationship description of the triple structure for joint representation learning of knowledge and balances the contribution of the triple structure and text description in the process of vector learning. First, we adopt the TransE model to learn the structural vector representations of entities and relations, and then use a CNN model to encode the entity description to obtain the text representation of the entity. To semantically encode the relation descriptions, we designed an Attention-Bi-LSTM text encoder, which introduces an attention mechanism into the Bi-LSTM model to calculate the semantic relevance between each word in the sentence and different relations. In addition, we also introduce position features into word features in order to better encode word order information. Finally, we define a joint evaluation function to learn the joint representation of structural and textual representations. The experiments show that compared with the baseline methods, our model achieves the best performance on both Mean Rank and Hits@10 metrics. The accuracy of the triple classification task on the FB15K dataset reached 93.2%.

1. Introduction

Knowledge graphs (KGs) adopt intuitive triplets to represent entities in the objective world and the relationships between them, storing a large number of facts in the form of triplets (head entity, relationship and tail entity). KGs were first applied to improve the capabilities of search engines. Subsequently, they have shown great application value in assisting intelligent question answering, natural language processing, big data analysis, recommendation computing and interpretable artificial intelligence [1,2,3]. Among them, knowledge representation is the foundation of these applications. However, due to the continuous accumulation of knowledge, the scale of KGs has expanded very quickly, and the forms of knowledge have become more and more diverse. The disadvantages of knowledge representation in traditional forms are becoming more and more obvious, such as the difficulty of reasoning about the semantic relationship between entities, the serious problem of data sparsity, the high complexity of calculation and the difficulty of applying it to large KGs. Knowledge representation learning [4,5,6] aims to solve the sparsity problem of KGs and improve their internal integrity.

Knowledge representation learning studies the vector representation of the research object in the low-dimensional vector space through matrix projection and explores the relationship between the embedded objects in the low-dimensional space. Representation learning for KGs aims to learn the representation of entities and relationships in KGs in low-dimensional vector space, which retains the semantic information of entities and relationships from various dimensions [7]. By calculating the semantic similarity between vectors, the internal structural association of KGs is further mined, and the overall quality of KGs is improved on this basis [8]. By representing entities and relationships, representation learning can solve problems such as data sparsity and knowledge reasoning difficulties faced by traditional representation methods, and is widely applied to downstream tasks related to KGs, such as knowledge acquisition, entity disambiguation, relation extraction and KG completion [9,10].

Similar to the idea of unsupervised semantic-aware-based graph representation learning [11], text information can provide rich semantic resources for graph representation and play an important auxiliary role in the representation learning optimization of a graph representation model. Therefore, recent research on KG representation learning research focuses on how to use the internal information of a knowledge graph to optimize the representation of knowledge. Most KG embedding methods only consider the triple structure in KGs, which ignores the additional information related to triples, especially text description information. Text description information is generally divided into entity description and relationship description. Entity description is defined as a text that describes a topic and an entity, often as a supplement to describe an entity from different aspects. The relation description is defined as a sentence containing triple entity pairs $(h,t)$, which can replace the direct relation $r$ in the triple group $(h,r,t)$ to express the relationship between entity pairs $(h,t)$. The distant supervision hypothesis [12] holds that if two entities have a certain relationship in a KG, then the sentences with these two entities usually contain this relationship, which indicates the correlation between the relationship description and the specific triple.

The combination of triplet structure and text description for KG embedding has become a hot topic in current research. Based on the triple structure, the entities and relationships in the KG are represented as vectors in the low-dimensional semantic space. The semantic information of entities and relationships is mined based on text description and mapped to the same semantic space. The Description-embodied Knowledge Representation (DKRL) [13] method integrates entity description information into triple embedding, and uses the Continuous Bag-of-Words [14] (CBOW) and a convolutional neural network (CNN) to encode entity descriptions. However, DKRL only takes some high-frequency words in the entity description as input, and only focuses on some high-frequency words and encodes the text representation of the entity as input, which is likely to cause semantic loss. Joint (Joint Representation Learning of Text and Knowledge) [15] proposed a joint representation learning model based on text and knowledge. It constructs a relationship text description by complex alignment of entities and relationships and uses a CNN to encode the relationship text to obtain the text representation of the relationship. However, when encoding the relationship description, Joint not only does not consider the word order information, but also ignores the difference in the degree of influence of each word on the relationship in the sentence. In addition, the DKRL model only considers the fusion entity description information, and the Joint model only considers the fusion relationship description information, both of which only use a text description from one perspective to enrich the representation of entities or relationships.

In addition, some works [13,16,17,18,19,20] introduced text description to improve the representation of KGs, but these methods still have shortcomings: (1) lack of joint representation learning framework based on triple structure and text description information, which effectively enhances the semantic representation of entities and relationships at the same time; (2) when encoding the text description information, the word order dependency cannot be effectively encoded; (3) the semantic importance of different words in text description sentences is not distinguished.

In order to solve the above problems, this paper proposes a KG representation model with a symmetric architecture called Joint Knowledge Representation Learning of Text Description and Knowledge Graph (JKRL). It models the entity description and relationship description on a triple structure for joint representation learning of knowledge and balances the contribution of triple structure and text description in the vector learning process. First, we adopt the TransE model to learn the structural vector representations of entities and relations, and then use a CNN model to encode the entity description to obtain the text representation of the entity. In particular, due to the relationship description statement having a strong correlation with a specific triple, we design an Attention-Bi-LSTM text encoder when we encode the relationship description. The attention mechanism [21] is introduced into the Bi-LSTM [22] model to calculate the semantic relevance of each word and different relations in the sentence, so as to dynamically select the most relevant information from the relationship description to obtain the accurate relationship text representation. In addition, we consider the relative distance between the word and the head–tail entity in the relational description sentence and introduce the position feature into the word feature to better encode the word order information. According to the characteristics of triple structure, entity description and relationship description, the JKRL model uses different coding models to extract their respective deep semantics, and the overall model architecture is symmetrical.

The main contributions of this work can be summarized as follows:

• A KG representation method called Joint Knowledge Representation Learning of Text Description and Knowledge Graph is proposed, which jointly learns the representation of entities and relations based on both structural and textual information, effectively improving the representation of entities and relations.
• An Attention-Bi-LSTM text encoder is designed to encode textual descriptions, which extracts word order features better by introducing word position features and attention mechanisms, and dynamically selects the most relevant information in the textual descriptions by calculating the semantic relevance of words based on different relationships.
• Experimental results on the FB15k dataset demonstrate that the JKRL model has strong competitiveness compared with the baseline model in KG completion tasks and can improve the quality of the vector representation of entities and relationships.

2. Related Work

Judging from the existing KG representation models, these models mainly use two types of information to assist knowledge embedding. The first is the existing triplet information in the KG. Most of these models are directly based on a single triplet. The group is modeled and only the triple itself is used for embedding; The second is additional information related to the triple, such as structural information, text description and entity type. Because additional information can enhance the semantic information in the vector representation of entities and relations in triples, such models can often improve the quality of vector representation of entities and relations.

2.1. Knowledge Representation Model Based on Single Triples

The knowledge representation model based on a single triplet only uses the internal information of the triplet itself, including the translation distance model, semantic matching model and new neural network model.

2.1.1. Translational Distance Models

Translation distance models usually calculate the semantic similarity between vectors based on the distance score function through the relational translation operation and then use the distance between two entities to measure the rationality of the triplet facts. TransE [23] regards relation as a transformation from the head entity to tail entity and measures the distance between the sum of the head entity vector and relation vector to the tail entity vector based on the distance scoring function. TransH [24] improved this process by projecting entities and relations onto the hyperplane of relations. TransR [25] introduced independent vector space for entities and relations, and used the specific vector space of relations to process different relations and mine more semantic information. TransD [26] is a simplification of TransR, which decomposes the projection matrix in TransR and introduces the respective projection vectors of the head and tail entities. TransMS [27] uses nonlinear functions to translate semantics and obtains tail entity vectors by translating head entity vectors and relational vectors.

2.1.2. Semantic Matching Models

Semantic matching models are usually based on similarity functions to match the latent semantics of entity and relation embeddings to measure the credibility of triples, and such models often have high computational complexity. DistMult [28] restricts the relation matrix ${\mathit{M}}_r$ to be a diagonal matrix and proposes a simplified bilinear formulation $f_r\left(h,t\right)={\mathit{h}}^{T}diag\left({\mathit{M}}_r\right)\mathit{t}$ to capture the interaction of head and tail entity vectors in the same dimension. HolE [29] learns representations of entities and relations by introducing products of compressed tensors. An analogy in [30] focuses on multi-relational reasoning to model the analogy structure of relational data, and its scoring function is $f_r\left(h,t\right)={\mathit{h}}^T{\mathit{M}}_r\mathit{t}$, where the relational matrix ${\mathit{M}}_r$ is defined as a normal matrix for analogy reasoning. The ComplEx [31] model extends DistMult by introducing holographic embedding and complex embedding, embedding entities and relations into complex space.

2.1.3. Neural Network Models

The neural network model based on deep learning is still based on the triple fact itself and optimizes the representation of entities and relationships by introducing neural network modules. ConvE [32] is the first model to use a CNN to complete KG embedding, and it is the simplest multi-layer convolutional network structure among all neural network models. ConvKB [33] uses 1D convolutions to preserve the translational properties of TransE and is able to capture the global relationship between entities. Experiments prove that ConvKB exhibits good link prediction performance and can be extended to other network data for application.

2.2. Knowledge Representation Model Fused with Additional Information

Representation learning based on a single triplet does not incorporate additional information about triplets to improve the representation ability of the model. The additional information is multi-source and can be divided into two categories. The first category is the information derived from the triple itself, which is included in the KG, such as structural information, entity type, etc., and in the second category the tuples are related but the sources of information are not inside the triples, such as entity descriptions, images associated with entities, etc.

2.2.1. Graph Structure Information

The graph structure information of a knowledge map can be regarded as being composed of the relationship path between entity neighborhood structure and entity pair. Entity neighborhood structure information includes neighborhood node information and neighborhood edge information. Relational path information refers to the sequence composed of entity nodes and the relationship edge between the entity pair without a direct relationship. Lin et al. [34] integrated the relationship path information based on the research of TransE, so that it can solve complex relationship problems and improve the effect of TransE on knowledge representation. The Graph Aware Knowledge Embedding (GAKE) model [35] extracts entity features from contextual information based on the word vector algorithm and applies them to downstream tasks, proving that the neighborhood structure information of entities can enhance the semantic representation of target entities.

2.2.2. Entity Description Information

Entity description information refers to text or images related to entities, and this article mainly refers to text descriptions related to entities. Text-Enhanced Representation Learning for Knowledge Graph (TEKE) [36] adopted a network structure in which entities and entity description information are combined, and combines the vector representation of entities based on text descriptions with the translation distance model to improve representation of the entity. DKRL [13] used two models of CBOW and CNN to encode the entity description to obtain the text vector representation of the entity, and improved the scoring function of the TransE model. Through the interaction between the triple structure and the entity description, the translation of the text vector of the entity in the relational space is realized. Joint [15] adopted a joint representation learning based on text and knowledge. Through a complex alignment of entity and relationship, the relational text description was constructed. Jointly [20] is a joint presentation framework based on the gating mechanism, which utilizes both structural information and textual information of entities and integrates the representation of structure and text into a unified architecture through the gating mechanism. An et al. [37] proposed an accurate text-enhanced KG representation learning method that enhances representation by utilizing entity description and relational description. In addition, an interactive attention mechanism based on relational description and entity description was proposed to learn more accurate text representation, so as to further improve the representation ability of KGs. Yao et al. [38] took the entity description and relation description of triples as input, introduced the pre-trained language model Bert [39] to learn rich language information from a large number of texts and highlighted the most important words associated with triples. Wang et al. [40] proposed a structure-augmented text representation model to tackle the link prediction task for KG completion. This model first applied a Siamese-style textual encoder to a triple for two contextualized representations. Then, based on the two representations, a scoring module was proposed where two parallel scoring strategies are used to learn both contextualized and structured knowledge. Shen et al. [41] proposed to jointly embed the semantics in the natural language description of the knowledge triplets with their structure information. This method embeds KGs via fine tuning pre-trained language models with respect to a probabilistic structured loss, where the forward pass of the language models captures semantics and the loss reconstructs structures. In addition, a large number of studies [42,43,44,45,46] have improved the representation of KGs by introducing text description and achieved good results.

2.2.3. Other Information

In addition to the above two mainstream and triple-related additional information, researchers try to integrate other information from different perspectives to improve the knowledge representation ability of the model. Xie et al. [47] combined the entity type and the triple structure, introduced the projection matrix of the entity type to map the head and tail entities to the relationship vector space, and then performed translation operations with the relationship vector in the relationship vector space, effectively solving the multi-type of entities and hierarchy issues. Esteban et al. [48] proposed to integrate the timeliness of quadruple facts to express KGs, which has a high application prospect for representing dynamic KGs. Guan et al. [49] incorporated the relation fact of n-element and represented it as a set of role–value key–value pairs. In addition, there is a Hyper-Relational KG Embedding model (HINGE) [50] that incorporates hyper-relational facts, where a hyper-relational fact contains a basic triple $(h,r,t)$ and a set of associated key–value pairs $(k_i,v_i)$.

3. Joint Knowledge Representation Learning of Text Description and Knowledge Graph

3.1. Symbols and Definitions

The KG is represented as $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$, where $\mathcal{E}$ denotes the entity set and $\mathcal{R}$ denotes the relation set. Every triple $(h,r,t)\in \mathcal{T}$ in the KG satisfies $h,t\in \mathcal{E}$, $r\in \mathcal{R}$. In the JKRL model, for a triplet $(h,r,t)$, the vector representation based on the triplet structure of the knowledge map is called a structural vector, which is denoted as ${\mathit{h}}_s$, ${\mathit{r}}_s$, ${\mathit{t}}_s$; the vector representation based on the text description is called text vectors, denoted as ${\mathit{h}}_d$, ${\mathit{t}}_d,{\mathit{r}}_d$. The symbol of the text description sentence is $s$, and the text description sentence set $T=\{s_1,s_2,\dots ,s_n\}$. Each sentence is a sequence of words, denoted as $s=\{x_1,x_2,\dots ,x_n\}$, where ${\mathit{x}}_i$ represents the embedding of word vectors. Entities and relations are embedded in a continuous vector space ${\mathbb{R}}^k$. The meanings of the specific symbols are shown in

Table 1. Symbols used in the JKRL model.

Symbol	Description
$h,r,t$	Head entity, relation entity and tail entity
${\mathit{h}}_s,{\mathit{r}}_s,{\mathit{t}}_s$	Structure vector of head entity, relation and tail entity
${\mathit{h}}_d,{\mathit{r}}_d,{\mathit{t}}_d$	Text vectors of head entity, relation and tail entity
${\mathit{x}}_i$	Word vector
${\alpha }_i\left(r\right)$	The attention weight of word $i$ under relation $r$
$s$	Text description statement
$T$	Text description statement set

.

3.2. The Overall Architecture of the Model

The JKRL model is a joint representation learning framework that integrates text description information of entities and relationships with KGs. The overall architecture is shown in Figure 1. The model consists of three knowledge representation learning parts, which are the representation learning of triple structure, representation learning of entity description and representation learning of relation description.

• Representation learning of triple structure (blue area): Learns the structural representation of entities and relationships from the perspective of KG. Based on the universality of the representation learning framework, a simple and efficient TransE model is used to embed the structure vector of the head and tail entities and the structure vector of the relationship, which are denoted as ${\mathit{h}}_s$, ${\mathit{t}}_s,{\mathit{r}}_s$.
• Representation learning of entity description (green area): A CNN model is used to encode the entity description of the triple $(h,r,t)$ to obtain the text representation of the head entity and tail entity, which are denoted as ${\mathit{h}}_d\mathrm{a}\mathrm{n}\mathrm{d}{\mathit{t}}_d$.
• Representation learning of relation description (yellow area): The relation description of the triple $(h,r,t)$ refers to the text containing the entity pair $(h,t)$. The Attention-Bi-LSTM text encoder is designed to mine the semantic information in the relation description to obtain the text representation of the relation, which is denoted as ${\mathit{r}}_d$.

The JKRL model performs joint representation learning based on structural representation and text representation. We define a joint evaluation function and perform vector representation to minimize the loss of training entities and relationships. The loss function of the JKRL model is defined as follows:

$${\mathcal{L}}_{\mathsf{\Theta }}(\mathcal{G},\mathcal{D})={\mathcal{L}}_{{\mathsf{\Theta }}_{E_S}{\mathsf{\Theta }}_{R_S}}\left(\mathcal{G}\right)+\alpha {\mathcal{L}}_{{\mathsf{\Theta }}_{E_D}}(\mathcal{G},\mathcal{D})+\beta {\mathcal{L}}_{{\mathsf{\Theta }}_{R_{\mathcal{D}}}}(\mathcal{G},\mathcal{D})+\lambda {‖\mathsf{\Theta }‖}_2$$

(1)

In the above formula, ${‖\mathsf{\Theta }‖}_2$ is the L2 regularization of the parameter $\mathsf{\Theta }$, the parameter set $\mathsf{\Theta }=\{{\mathsf{\Theta }}_{E_S},{\mathsf{\Theta }}_{E_D},{\mathsf{\Theta }}_{R_S},{\mathsf{\Theta }}_{R_D}\}$, ${\mathsf{\Theta }}_{E_S}$ represents the parameter set of the learning entity structure representation, ${\mathsf{\Theta }}_{E_D}$ represents the parameter set of the learning entity text representation, ${\mathsf{\Theta }}_{R_S}$ represents the parameter set of the learning relationship structure representation and ${\mathsf{\Theta }}_{R_D}$ represents the parameter set of the learning relationship text representation. ${\mathcal{L}}_{{\mathsf{\Theta }}_{E_S}{\mathsf{\Theta }}_{R_S}}\left(\mathcal{G}\right)$ is the loss function of representation learning based on triple structure, ${\mathcal{L}}_{{\mathsf{\Theta }}_{E_D}}(\mathcal{G},\mathcal{D})$ is the loss function of representation learning based on entity description and ${\mathcal{L}}_{{\mathsf{\Theta }}_{R_{\mathcal{D}}}}(\mathcal{G},\mathcal{D})$ is the loss function of representation learning based on relation description. The $\mathsf{\alpha }$ and $\mathsf{\beta }$ in the equation are to balance the role of triple structure and text description information in vector joint representation learning. Formula (1) can balance the contribution of the two vector representations in the learning process.

3.3. Representation Learning of Triple Structure

Traditional KG representation is to generate vectors in low-dimensional continuous space based on triple structure by representation learning technology and construct scoring functions to evaluate the reliability of triple facts. TransE is a classic KG representation learning model, which has been widely used and verified in many studies combined with text descriptions, such as DKRL [13], Joint [15], Jointly [20], etc. Since the JKRL model needs to jointly learn the representation of triplets and text descriptions and uses deep neural networks such as CNN and Attention-Bi-LSTM, the model has many trainable parameters and has high space complexity. Therefore, in the representation learning of triple structure, we employ TransE, a simple but effective model, to learn structured vector representations of entities and relations. In practice, more complex models, such as TransH, TransR and DisMult [24,25,28], may overfit or be too complex to train.

TransE assumes that the vector space dimensions of entities and relations are consistent and initializes the triple $(h,r,t)$ to obtain $({\mathit{h}}_s,{\mathit{r}}_s,{\mathit{t}}_s$). Then, ${\mathit{r}}_{ht}$ is set to be the translation of the head entity vector ${\mathit{h}}_s$ to the tail entity vector ${\mathit{t}}_s$:

$${\mathit{r}}_{ht}={\mathit{t}}_s-{\mathit{h}}_s$$

(2)

Since triple embedding $({\mathit{h}}_s,{\mathit{r}}_s,{\mathit{t}}_s$) denotes an explicit relation vector ${\mathit{r}}_s$ between ${\mathit{h}}_s$ and ${\mathit{t}}_s$, the triple scoring function is defined as follows:

$${\phi }_r(h,t)={‖{\mathit{r}}_{ht}-{\mathit{r}}_s‖}_2={‖({\mathit{t}}_s-{\mathit{h}}_s)-{\mathit{r}}_s‖}_2$$

(3)

The above formula shows that each triple $(h,r,t)$ existing in the KG should satisfy ${\mathit{h}}_s+{\mathit{r}}_{\mathit{s}}={\mathit{t}}_s$ in the ideal state, and the result of the triple ${\mathit{h}}_s+{\mathit{r}}_s$ that does not exist or is incorrect in the KG should be far away from ${\mathit{t}}_s$. The structural vectors ${\mathit{h}}_s$, ${\mathit{t}}_s$ and ${\mathit{r}}_s$ of entities and relationships are obtained by training the model. Therefore, the scoring function for all triples is defined as

$${\mathcal{L}}_{{\mathsf{\Theta }}_{E_S}{\mathsf{\Theta }}_{R_S}}\left(\mathcal{G}\right)={\sum }_{(h,r,t)\in S}{\sum }_{(h^{\prime },r^{\prime },t^{\prime })\in S^{\prime }}{\left[{\phi }_r\right(h,t)-{\phi }_{r^{\prime }}(h^{\prime },t^{\prime })+\mu ]}_{+}$$

(4)

where S represents the set of positive triples in the KG, $S^{\prime }$ represents the set of negative triples after negative sampling, specifically represented as $S^{\prime }=\left\{\right(h^{\prime },r,t\left)\right\}\cup \left\{\right(h,r^{\prime },t\left)\right\}\cup \left\{\right(h,r,t^{\prime }\left)\right\}$, $h^{\prime }\in \mathcal{E}$ represents the head entity obtained by sampling, $r^{\prime }\in \mathcal{R}$ represents the relationship obtained by sampling and $t^{\prime }\in \mathcal{E}$ represents the tail entity obtained by sampling.

3.4. Representation Learning of Entity Descriptions

The representation learning of entity description aims to extract the semantics related to the topic entity in the text by encoding the entity description statement to model the vector representation of the entity. After obtaining the text vector ${\mathit{h}}_d$ of the head entity and the text vector ${\mathit{t}}_d$ of the tail entity, the scoring function of each triple is defined as follows:

$${\sigma }_r(h,t)={‖{\mathit{t}}_d-{\mathit{h}}_d-{\mathit{r}}_s‖}_2$$

(5)

For all triples in the KG, the loss function of the module is defined as

$${\mathcal{L}}_{{\mathsf{\Theta }}_{E_D}}(\mathcal{G},\mathcal{D})={\sum }_{(h,r,t)\in S}{\sum }_{(h^{\prime },r,t^{\prime })\in S^{\prime }}{\left[{\sigma }_r\right(h,t)-{\sigma }_r(h^{\prime },t^{\prime })+\eta ]}_{+}$$

(6)

where $\eta$ is an interval parameter greater than 0. $S$ is a positive triple set, $S^{\prime }$ is a negative triple set and $S^{\prime }=\left\{\right(h^{\prime },r,t\left)\right\}\cup \left\{\right(h,r,t^{\prime }\left)\right\}$.

The entity description text encoding process is shown in Figure 2. DKRL only encodes some high-frequency words in the entity description, which easily causes semantic information loss. Since the entity description text may present an entity from various aspects and is not sensitive to the sequence information of the text, a CNN can effectively capture local features in the entity description text by sliding on the text sequence through the convolutional kernel and has the advantage of parallel computing with high computational efficiency, which is suitable for processing large-scale text data. Therefore, we adopt a CNN model to encode the text representation of the entity, and mine the deep semantic information hidden in the word order inside the entity description. The specific steps are as follows:

• Input: Firstly, all stop words and punctuations in the entity text description are deleted to construct the expected vocabulary for training. Then, the entity annotation tool is used to automatically annotate the entities in $\mathcal{G}$ to obtain the annotated text, forming the entity text description $s=\{{\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_n\}$ related to a specific triple. Finally, the word2vec tool is used to train the word vector to correspond to each word in the text description to form the input vector $\mathit{X}=({\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_n)$.
• Convolution layer: The input and output of the convolution layer are represented by $\mathit{X}$ and $\mathit{C}$, respectively. Because the length of the text description statement is different, the length of the input vector $\mathit{X}=({\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_n)$ is inconsistent. After setting the maximum sentence length $L$ for the convolution operation, if the sentence length is less than $L$, the end of the embedding vector of the sentence is filled with 0, and if the sentence length is greater than $L$, the redundant words are removed. We need to add 0 to the end of the embedded vector of the sentence. Then, the input vector $\mathit{X}$ is processed based on the sliding window mechanism. We define the size of the window as $k$, set the step size of the sliding window as 1, and the content of the sliding window is ${\mathit{x}}_{i:i+k-1}=[{\mathit{x}}_i,{\mathit{x}}_{i+1},\dots ,{\mathit{x}}_{i+k-1}]$. The corresponding output vector ${\mathit{c}}_i$ is extracted by a convolution operation. The specific formula of this process is defined as follows:

  $${\mathit{c}}_i={\sigma }_c({\mathit{w}}_c{\cdot \mathit{x}}_{i:i+k-1}+{\mathit{b}}_i)$$

  (7)

  where ${\mathit{w}}_c\in R^{m\ast kn}$ represents the convolution kernel, m is the dimension of the output vector, n represents the dimension of the input vector, ${\mathit{b}}_i$ represents the bias term, and ${\sigma }_c$ represents the activation function.
• Pooling layer: In order to reduce the number of parameters of the CNN encoder and the influence of noise, the feature vectors’ output by the convolutional layer needs to be pooled. We use the maximum pooling layer to optimize the semantic features in the feature vectors. It is divided into multiple non-overlapping windows, and the maximum value is selected in each window to form a new output vector $\mathit{p}$ in order to obtain the most significant features in each dimension:

  $$\mathit{p}=max({\mathit{c}}_i,{\mathit{c}}_{i+1},\dots ,{\mathit{c}}_{i+m-1})$$

  (8)
• Output layer: The text vector of the entity is obtained by multiplying the vector matrix of the input vector. The text vector form of the entity is defined as

  $${\mathit{e}}_d={\mathit{w}}_0\cdot \mathit{p}+b_0$$

  (9)

  where ${\mathit{w}}_0$ is the parameter matrix and $b_0$ is the bias term, and finally the head and tail entity text vectors ${\mathit{h}}_d$ and ${\mathit{t}}_d$ are obtained.

3.5. Representation Learning of Relation Description

Relation description reference sentences refer to sentences containing triple entity pairs $(h,t)$. The representation learning of relation description aims to reduce the noise in relation description sentences and obtain the vector representation of relation based on text description. For each triple, the scoring function is defined as the minimum distance between ${\mathit{r}}_s$ and ${\mathit{r}}_d$:

$${\psi }_r\left(s\right)={‖{\mathit{r}}_s-{\mathit{r}}_d‖}_2$$

(10)

The loss function for all relation description statement sets is defined as follows:

$${\mathcal{L}}_{{\mathsf{\Theta }}_{R_{\mathcal{D}}}}(\mathcal{G},\mathcal{D})={\sum }_{s\in T}{\sum }_{r^{\prime }\ne r}{\left[{\psi }_r\right(s)+{\psi }_{r^{\prime }}(s)+\gamma ]}_{+}$$

(11)

where $\gamma$ is an interval parameter greater than 0, and T represents the set of text description statements.

Since the relationship description information is only one sentence, the text length is short and contains less information. After comprehensive consideration of data characteristics, computing resources, dataset size and time sensitivity, we finally choose Bi-LSTM, which has a moderate computational complexity and number of parameters, as the basic model for text encoder design among commonly used text modeling models such as CNN, RNN and Transformer [22,51,52]. The Bi-LSTM model can remember the whole semantic information of the sentence through the gating mechanism when the sentence length is short, and the problem of forgetting will not occur.

Because the text relationship hidden in the relationship description has a strong correlation with a specific triple, it is necessary to retain the word order features and the dependencies between words in the statement when modeling the relationship description, so as to extract the deep semantic features of the relationship description. When modeling text description, most methods not only encode the word order features in relational description poorly, but also ignore the semantic relevance of the word and the relation in the triplet, because the text description may present an entity from various aspects, and the relationship may only be closely related to part of the statement. In the JKRL model, we designed an Attention-Bi-LSTM text encoder, as shown in Figure 3. On the basis of Bi-LSTM, the attention mechanism layer is introduced. According to the different relationships in the corresponding triples, the most relevant word semantics are dynamically selected from the relationship description for coding. The Attention-Bi-LSTM is used to model the relationship description as follows:

• The relationship description statement is preprocessed and represented by word embedding. The feature vector of each word is obtained by word2vec training. In addition, the relative distance between each word and the head and tail entities in the sentence is calculated.
• In the Attention-Bi-LSTM encoder, the Bi-LSTM hidden layer performs forward and backward LSTM encoding on the input vector.
• In the attention layer of Attention-Bi-LSTM, the corresponding attention weight is calculated according to the correlation degree between each word and the triplet relationship, and finally the text vector ${\mathit{r}}_d$ of the relationship is obtained in the output layer.

• Word embedding representation

The input vector of the relation description $s=\{{\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_n\}$ is composed of all word vectors in the sentence. For each word, the word feature (WF) and the position feature (PF) together form the word vector.

Word2vec [53] is used for embedding learning. Word2vec encodes the context information of words in the corpus. The word embedding obtained can be directly used as the representation of word features:

$$WF=\{{\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_n\}$$

(12)

Relational description sentences contain head and tail entities, whose semantic information can enhance the expression of relational semantics in related triples. The position feature is defined as the relative distance between each word in the sentence and the head and tail entities. In order to measure the distance between a word and the head–tail entity, the position feature [53] $PF$ is defined as follows:

$$PF=\{\left[d_{1h},d_{1t}\right],\left[d_{2h},d_{2t}\right],\dots ,\left[d_{nh},d_{nt}\right]\}$$

(13)

where $d_{ih}$ denotes the relative distance between the $i$ word and the head entity $h$, and $d_{it}$ denotes the relative distance between the $i$ word and the tail entity $t$. For example, for the relational description statement “Putin is the president of Russia”, the relative distance between the word “president” and the head entity “Putin” is 3, and the relative distance between the word “president” and the tail entity “Russia” is −2. The relative distance between the word and its left entity is recorded as positive distance, and the distance between the word and its right entity is recorded as negative distance. Finally, we splice all the word features and their corresponding position features in the sentence to obtain the vector representation $\mathit{s}$ of the sentence as the input vector of the Bi-LSTM encoder.

$$\mathit{s}=\{\left[{\mathit{x}}_1,d_{1h},d_{1t}\right],\left[{\mathit{x}}_2,d_{2h},d_{2t}\right],\dots ,\left[{\mathit{x}}_n,d_{nh},d_{nt}\right]\}$$

(14)

2.

Bi-LSTM coding

A recurrent neural network (RNN) is used to process sequence information. Due to the need to process both current information and historical information when dealing with long-term dependencies, a RNN has gradient disappearance and gradient explosion problems [54]. LSTM is an enhanced neural network based on RNN, which realizes the interaction between short-term memory and long-term memory through a gating mechanism and solves the problem that RNN cannot remember long-term historical input information. The LSTM hidden layer contains the memory cell, input gate ${\mathit{i}}_t$, forgetting gate ${\mathit{f}}_t$ and output gate ${\mathit{o}}_t$; ${\mathit{s}}_t$ is set to be the input at time t, ${\mathit{c}}_{t-1}$ is the output of the memory cell at time t − 1 and ${\mathit{h}}_{t-1}$ is the output of the hidden layer at time t − 1. Therefore, for the hidden state ${\mathit{h}}_t$ at time t, the calculation process of the LSTM hidden layer is as follows:

$${\mathit{i}}_t=\sigma ({\mathit{W}}_{si}{\mathit{s}}_t+{\mathit{W}}_{hi}{\mathit{h}}_{t-1}+{\mathit{W}}_{ci}{\mathit{c}}_{t-1}+{\mathit{b}}_i)$$

(15)

$${\mathit{f}}_t=\sigma ({\mathit{W}}_{sf}{\mathit{s}}_t+{\mathit{W}}_{hf}{\mathit{h}}_{t-1}+{\mathit{W}}_{cf}{\mathit{c}}_{t-1}+{\mathit{b}}_f)$$

(16)

$${\mathit{o}}_t=\sigma ({\mathit{W}}_{so}{\mathit{s}}_t+{\mathit{W}}_{ho}{\mathit{h}}_{t-1}+{\mathit{W}}_{co}{\mathit{c}}_t+{\mathit{b}}_o)$$

(17)

$$\widetilde{{\mathit{c}}_t}=tanh({\mathit{W}}_{sc}{s}_t+{\mathit{W}}_{hc}{\mathit{h}}_{t-1}+{\mathit{b}}_c)$$

(18)

$${\mathit{c}}_t={\mathit{f}}_t{\mathit{c}}_{t-1}+{\mathit{i}}_t\widetilde{{\mathit{c}}_t}$$

(19)

$${\mathit{h}}_t={\mathit{o}}_{t}tanh\left({\mathit{c}}_t\right)$$

(20)

where, ${\mathit{W}}_{si}$, ${\mathit{W}}_{hi}$, ${\mathit{W}}_{ci}$, ${\mathit{W}}_{sf}$, ${\mathit{W}}_{hf}$, ${\mathit{W}}_{cf}$, ${\mathit{W}}_{so}$, ${\mathit{W}}_{ho}$, ${\mathit{W}}_{co}$, ${\mathit{W}}_{sc}$ and ${\mathit{W}}_{hc}$ are weight matrices, ${\mathit{b}}_i$, ${\mathit{b}}_f$, ${\mathit{b}}_o$ and ${\mathit{b}}_c$ are bias vectors and $\sigma$ is the Sigmoid function.

Since LSTM cannot reverse the information from back to front, the Bi-LSTM neural network contains two LSTM hidden layers: one is responsible for the forward transmission of information, and the other is responsible for the reverse transmission of information. Finally, the two parts of information are combined to generate the final output of Bi-LSTM. For Bi-LSTM, each word vector is input into two LSTM hidden layers: ${\mathit{s}}_t$ is the embedding vector of the word input at time t, $h_{t-1}^f$ is the output of the previous moment of the forward LSTM and $h_{t+1}^b$ is the output of the previous moment of the backward LSTM. The output of the forward and backward LSTM at time t is as follows:

$${\mathit{h}}_t^f=H({\mathit{s}}_t,{\mathit{h}}_{t-1}^f,{\mathit{c}}_{t-1},{\mathit{b}}_{t-1})$$

(21)

$${\mathit{h}}_t^b=H({\mathit{s}}_t,{\mathit{h}}_{t+1}^f,{\mathit{c}}_{t-1},{\mathit{b}}_{t-1})$$

(22)

where $H(\dots )$ represents the encoding process of LSTM. Each output unit of Bi-LSTM is connected to the forward and backward LSTM hidden layers at this time. The output vector at time $t$ is as follows:

$${\mathit{z}}_t=\sigma ({\mathit{W}}_{hz}^f{\mathit{h}}_t^f+{\mathit{W}}_{hz}^b{\mathit{h}}_t^b+{\mathit{b}}_z)$$

(23)

where ${\mathit{W}}_{hz}^f$, ${\mathit{W}}_{hz}^b$ are weight matrices and ${\mathit{b}}_z$ is the bias vector.

3.

Attention mechanism

Relational description sentences contain head and tail entities whose semantic information can enhance the expression of relational semantics in related triples. However, since the relationship description statement contains multiple words, not all words have an effect on the semantic information of the modeling description statement. If a simple summary will introduce word-level noise, it is necessary to calculate the correlation between each word in the relationship description and its corresponding specific triple. We add an attention layer to the Bi-LSTM encoder, calculate the weight of each word and relationship in the sentence based on the attention mechanism and obtain the text vector ${\mathit{r}}_d$ of the relationship according to the weight.

For the word at position $i$ in the relation description, we define its attention weight to the given relation $r$ as ${\alpha }_i\left(r\right)$:

$$e_i\left(r\right)={\mathit{V}}_a^{T}tanh({\mathit{W}}_a{\mathit{z}}_i+{\mathit{U}}_a\mathit{r})$$

(24)

$${\alpha }_i\left(r\right)=softmax\left(e_i\left(r\right)\right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(e_i\right(r\left)\right)}{{\sum }_{j=1}^n\mathrm{e}\mathrm{x}\mathrm{p}\left(e_j\right(r\left)\right)}$$

(25)

where ${\mathit{z}}_i$ represents the output of the Bi-LSTM hidden layer at position $i$ and ${\mathit{W}}_a$, ${\mathit{U}}_a$ and ${\mathit{V}}_a$ are parameter matrices. The output state $\mathit{c}$ of the attention mechanism layer is calculated as follows:

$$\mathit{c}={\sum }_{i=1}^n{\alpha }_i\left(r\right)\cdot {\mathit{z}}_i$$

(26)

Finally, the output vector $\mathit{c}$ of the attention mechanism layer is nonlinearly transformed through the activation function to obtain the relational text vector ${\mathit{r}}_d$:

$$\mathit{c}{\mathit{r}}_d=\sigma \left(\mathit{c}\right)$$

(27)

4. Experiment and Result Analysis

4.1. Experimental Environment

The hardware environment used in this experiment is Intel Core i5-9300HF, 16 GB memory, GTX1660Ti.

4.2. Dataset

In order to evaluate the effectiveness of our proposed method, we used the most common dataset FB15K [23] in the KG completion task. FB15K is a subset of Freebase [55], and the text description corresponding to the entities in the dataset is publicly available. We chose the FB15K dataset as the experimental data because it is one of the most commonly used datasets in the field of KG representation learning and has been extensively researched and validated. The dataset contains multiple types of entities and contains multiple complex relationships, which can help us evaluate the performance of the model on different types of triples. In particular, the attention mechanism in A-BI-LSTM assigns weight dynamically according to different relations and word correlations, while the FB15K dataset contains a variety of complex relations which can better verify the role of the A-Bi-LSTM module. The statistical information of the dataset is shown in

Table 2. Statistics of FB15k dataset.

Dataset	Number of Entities	Relation Number	Training Set	Validation Set	Test Set
FB15k	14,904	1341	472,860	48,991	57,803

.

Entity description refers to the statement that contains the subject entity. We checked whether the entity in the dataset had a corresponding description, filtered a total of 47 entities that were not sufficiently described and entities with a text description length of less than 3, and deleted all triples in the dataset containing the relevant entity. After preprocessing, the average word count of entity descriptions is 69, the shortest description contains 3 words, and the longest description contains 343 words.

Relational description refers to sentences containing entity pairs. Corpus NYT [15] is an existing work. Sentences were selected from New York Times articles to align with FB15K for joint learning. To ensure alignment accuracy, we only considered those sentences with anchor text linking to the entities in FB15K. We extracted 876,227 sentences containing both head and tail entities in FB15K triples and annotated them with the corresponding relations in triples. The sentences were labeled with 29,252 FB15K triples, including 629 relations and 5244 entities. Figure 4 shows an instance of a triplet in the dataset that contains both the relationship description and the entity description.

4.3. Evaluation Indicators

The KG completion task can be divided into two parts: the link prediction task and triple classification task. The link prediction task can be further divided into the entity prediction task and relationship prediction task. Link prediction tasks and triplet classification tasks have been widely used to evaluate the quality of knowledge representation models.

1.

Link prediction task

The link prediction task is divided into the entity prediction task and relationship prediction task. The purpose of the entity prediction task is to predict the lack of head entities or tail entities in the triple, that is, given $(h,r)$ prediction $t$ or given $(r,t)$ prediction $h$; the purpose of the relationship prediction task is to predict the missing relationship in the triple, that is, to predict $r$ for a given $(h,t)$.

In order to measure the performance of the model, for a triple which is to be tested, that is, an entity or relationship has partial content missing, each entity or relationship in the KG $\mathcal{G}$ is used to fill the missing content to obtain a candidate triple. Because the larger the score, the lower the credibility of the candidate triple, then the evaluation function $‖{\mathit{h}}_s+{\mathit{r}}_s-{\mathit{t}}_s‖$ is used to calculate the score corresponding to each candidate triple and arrange it in ascending order. Finally, the established triple facts are ranked and the corresponding Mean Rank (MR) value and Hits@N value are calculated.

Mean Rank refers to the average value of the correct entity or relationship ranking position in the test set. For the head entity prediction task, the trained model traverses the candidate entities to replace the head entity $h$ of the triple $(h,r,t)$ in the test set and ranks the generated triples. The score is calculated for each triplet generated by replacing all candidate entities, the scores are sorted in descending order and the correct answer is recorded. The specific calculation formula of Mean Rank according to this process is as follows:

$$MR=\frac{1}{\left|Q\right|}\sum _{i=1}^{\left|Q\right|}{rank}_i$$

(28)

where $\left|Q\right|$ denotes the number of triples in the test set, and ${rank}_i$ denotes the ranking position of the $i$ th test triple.

Hits@N refers to the proportion of the ranking position of the correct answer of the test triple in the top N. Therefore, it can be seen that the smaller the MR value is, the higher the ranking position of the correct triples is. The larger the Hits@N value is, the higher the proportion of the ranking position of all correct triples in the top N position is. This shows that the higher the predictive ability of the model for correct answers, the better the knowledge map completion performance of the model. Lower MR values and higher Hits@N indicate that the model has better prediction performance. We call this evaluation setting “Raw”. Since a false predicted triplet may also exist in knowledge graphs, it should be regard as a valid triplet. Hence, we should remove the false predicted triplets included in training, validation and test sets before ranking. We call this evaluation setting “Filter”. The evaluation results are reported under these two settings.

2.

Triple classification task

The triple classification task is mainly to distinguish the authenticity of a triple $(h,r,t)$, which is usually considered as a binary classification problem. The triple $(h,r,t)$ in the test set is scored by the model. If the score of the triple is higher than the set threshold ${\delta }_r$, the triple is considered to be correct, otherwise it is considered to be invalid. The threshold ${\delta }_r$ is related to the relationship in its corresponding triples, and ${\delta }_r$ is determined by maximizing the accuracy of triple classification corresponding to different relationships in the validation set.

The triplet classification task usually uses accuracy as the evaluation index, which is defined as follows:

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}\mathrm{P}=\frac{\mathrm{T}\mathrm{h}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{e}\mathrm{d}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{s}}{\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{s}}$$

(29)

4.4. Baseline Model

In order to prove the effectiveness of our model, we compare JKRL with the following two models: (1) translation distance model, including TransE [23], TransH [24], TransR [25] and TransD [26]; (2) Joint representation learning models integrating text description and structure, including DKRL [13], Joint [15], Jointly (LSTM) [20] and Jointly (A-LSTM) [20]. In order to analyze the effect of the model more comprehensively, we designed three variant models of JKRL(CNN+Bi-LSTM), JKRL(A-Bi-LSTM) and JKRL(CNN+A-Bi-LSTM).

TransE [23]: TransE is the first time a method has been proposed for training KG embedding based on the translation distance function, and it is also one of the most classic and widely used KG embedding methods.

TransH [24]: TransH relates each relationship to a relationship-specific hyperplane to alleviate the modeling problem of complex relationships.

TransR [25]: TransR believes that entities based on different relationships should exhibit different attributes by introducing relationship-specific vector spaces to deal with different relationships to mine more semantic information.

TransD [26]: The TransD model decomposes the projection matrix in TransR and introduces the projection vectors of head and tail entities to effectively distinguish the differences between head and tail entities.

DKRL [13]: The model of entity description for KG representation learning is introduced, and two coding methods of the continuous word bag model and convolutional neural network are explored to extract the semantic information of entity description. However, DKRL only considers the fusion of entity description information to enrich the semantic information of entity embedding.

Joint [15]: a joint learning framework based on knowledge graph and relational representation using CNN to encode the relation description. However, Joint only improves the representation of the relationship based on the relationship description.

Jointly (Bi-LSTM) [20]: Jointly (Bi-LSTM) is a joint representation learning framework that integrates text descriptions. This model uses TransE to learn the structure representation of triples and uses Bi-LSTM to encode the entity description information to obtain the text representation of entities. Finally, the gating mechanism is applied to integrate the structure and text representation of entities as the final joint representation.

Jointly (A-Bi-LSTM) [20]: Based on Jointly (Bi-LSTM), the attention mechanism is introduced to capture the most relevant content of sentence level and relationship to model the text representation of entities.

JKRL(CNN+Bi-LSTM): When the JKRL model is learning the representation of relation description, the location feature in the word feature is deleted, and the Bi-LSTM excluding the attention layer is used for encoding.

JKRL(A-Bi-LSTM): A-Bi-LSTM is used to encode the representation learning of entity description and relationship description in the JKRL model. Specifically, in the representation learning of entity description, the relative distance between each word and the head or tail entity in the sentence is added to the word feature representation as a positional feature, and the attention weight between each word and the head or tail entity is calculated at the attention level to obtain the text vector of the entity.

JKRL(CNN+A-Bi-LSTM): This is the JKRL model proposed in this paper. A CNN is used for representation learning for entity description and A-Bi-LSTM is used for representation learning for relational description.

4.5. Experimental Settings

We limited the hyperparameter range of the JKRL model and determined the optimal hyperparameter setting of the model by minimizing the MR value on the verification set. The range of all hyperparameters in the experiment was set as follows: the learning rate of the KG structure vector ${\lambda }_{\mathcal{G}}\in \left\{\mathrm{0.01,0.001,0.0005}\right\}$, the learning rate of the text vector ${\lambda }_{\mathcal{D}}\in \left\{\mathrm{0.01,0.025,0.05}\right\}$, $\mathsf{\mu }$, $\mathsf{\eta }$, $\mathsf{\gamma }$ and the learning rate $\lambda$ were set to 1. The maximum sentence length $L\in \left\{\mathrm{80,90,100}\right\}$ when using CNN to perform convolution operations on entity description text. The value range of the balance parameters α and β was $\left\{\mathrm{0.01,0.001,0.0001}\right\}$, and the dimension range of all vectors $k\in \left\{\mathrm{50,100,150,200}\right\}$. After combining different parameters for grid search, the optimal configuration of the model was as follows: ${\lambda }_{\mathcal{G}}=0.001$, ${\lambda }_{\mathcal{D}}=0.025,L=90,\mathsf{\alpha }=0.001,\mathsf{\beta }=0.0001,k=100$.

In order to explore the influence of balance parameters $\mathsf{\alpha }$, $\mathsf{\beta }$ and vector dimension k on the final performance of the model, we visualized the changes in MR values of the verification set under different values of these hyperparameters under optimal configurations of other hyperparameters, as shown in Figure 5.

As can be seen from the figure, the model performance is optimal when the balance parameter $\alpha$ value is 0.001 and $\mathsf{\beta }$ value is 0.0001. Their function is to balance the contribution degree of the representation learning based on entity description and relation description to the triplet structure learning in the joint evaluation function so as to prevent the loss function of textual description from affecting the all-important representation of triples. When the dimension $k$ value of a vector is 100, the model performance is optimal, which belongs to a small dimension, indicating that the JKRL model can effectively improve the sparsity of knowledge graph data.

4.6. Analysis of Experimental Results

1.

Entity prediction task

Table 3. Entity prediction results.

Evaluating Indicator	Mean Rank		Hits@10
	Raw	Filter	Raw	Filter
TransE	243	125	0.349	0.471
TransH	212	87	0.457	0.644
TransR	198	77	0.482	0.687
TransD	194	81	0.534	0.773
DKRL	181	91	0.496	0.674
Jointly(Bi-LSTM)	179	90	0.493	0.697
Jointly(A-Bi-LSTM)	167	73	0.529	0.755
Joint	—	—	—	0.787
JKRL(CNN+Bi-LSTM)	162	78	0.538	0.786
JKRL(A-Bi-LSTM)	154	71	0.559	0.802
JKRL(CNN+A-Bi-LSTM)	148	67	0.574	0.813

shows the experimental results of the JKRL and baseline models in the entity prediction task on the FB15K dataset. Through the observation and analysis of the data in the table, we can draw the following conclusions:

• Joint representation learning models combining structure and text description are better than the Trans series models based on structural representation in Mean Rank and Hits@10, indicating that effectively integrating the text description can provide rich semantic resources for knowledge graph representation, thus improving entity and relationship representation. It plays an important auxiliary role in the optimization of knowledge graph representation learning.
• JKRL outperforms other baseline models that combine structure and text description for joint representation in Mean Rank and Hits@10, indicating that the JKRL model architecture can extract features from the entity description and relationship description, so as to effectively conduct joint representation learning based on structure and text description. Since DKRL and Jointly only consider the fusion of entity description information, and Joint only considers the fusion of relationship description information, they only use text descriptions to enrich the representation of entities or relationships from one perspective, resulting in limited improvement in representation. JKRL combines entity description and relation description to carry out joint representation learning of knowledge graphs. In addition, according to the characteristics of and differences in the two pieces of descriptive information, the corresponding coding model is designed to fully extract the deep semantic features in the text description, thus effectively improving the representation learning ability of knowledge graph.
• In particular, JKRL(A-Bi-LSTM) achieved better experimental results than Jointly(A-Bi-LSTM), which indicates that JKRL can better extract word order features by introducing word position features and using A-Bi-LSTM to encode relationship descriptions. It also shows that combining entity description and relationship description at the same time to improve the knowledge map representation learning is better than using unilateral text description information.
• The data index of JKRL(CNN+A-Bi-LSTM) is significantly better than that of JKRL(CNN+Bi-LSTM). The attention mechanism of JKRL(CNN+A-Bi-LSTM) is to dynamically allocate weights according to different relationships and word correlations. Because the FB15K dataset contains a variety of complex relationships, JKRL(A-Bi-LSTM) can play a better role, which shows the effectiveness of the Attention-Bi-LSTM encoder of the model.
• JKRL(A-Bi-LSTM) adopts the A-Bi-LSTM encoder for both entity description and relation description, and both introduce word location features. However, compared with JKRL(CNN+A-Bi-LSTM), the performance was not improved, and even decreased. This shows that the effect of A-Bi-LSTM in text feature extraction for entity description is not as good as CNN. Since the entity description text may present an entity from various aspects, and the position of the head and tail entities in the text has no direct influence on the semantic expression of the text, the introduction of the attention mechanism and location features in the A-Bi-LSTM encoder does not play an effective role in the representation of the entity description. Noise is even introduced to reduce the entity representation effect, which leads to the overall joint learning performance decline.

The relationship types in the KG can be divided into four categories: 1-1 relationship, 1-N relationship, N-1 relationship and N-N relationship. We further compared the prediction performance of the model under different relationship types.

Table 4. Entity prediction results under different relationship types.

Experiment	Head Entity Prediction				Tail Entity Prediction				Overall
Relation Classification	1-1	1-N	N-1	N-N	1-1	1-N	N-1	N-N
TransE	0.437	0.657	0.182	0.472	0.437	0.197	0.667	0.500	0.471
TransH	0.668	0.876	0.287	0.645	0.655	0.398	0.833	0.672	0.644
TransR	0.788	0.892	0.341	0.692	0.792	0.374	0.904	0.721	0.687
TransD	0.861	0.955	0.398	0.785	0.854	0.506	0.944	0.812	0.773
DKRL	—	—	—	—	—	—	—	—	0.674
Jointly(Bi-LSTM)	0.813	0.889	0.188	0.452	0.801	0.254	0.896	0.524	0.697
Jointly(A-Bi-LSTM)	0.838	0.951	0.211	0.479	0.830	0.308	0.947	0.531	0.755
Joint	0.827	0.891	0.450	0.807	0.817	0.577	0.874	0.828	0.787
JKRL(CNN+Bi-LSTM)	0.852	0.919	0.402	0.765	0.838	0.512	0.917	0.803	0.786
JKRL(A-Bi-LSTM)	0.867	0.935	0.411	0.776	0.848	0.519	0.938	0.818	0.802
JKRL(CNN+A-Bi-LSTM)	0.872	0.958	0.424	0.780	0.865	0.527	0.955	0.832	0.813

details the average Hits@10 values of the predicted head entity and the predicted tail entity under different relationship types.

Through the observation and analysis of the data in the table, we can draw the following conclusions:

• The joint representation model that combines structure and text description is superior to the Trans series model in overall entity prediction results. TransD enables the interaction between entities and relationships and also achieved good entity prediction performance. This shows that in addition to optimizing the knowledge graph representation based on the triple structure, the semantic representation of entities or relationships can be effectively enhanced by fusing the triple structure and text description, and the representation ability of the knowledge graph can be improved.
• Compared with the structure-based Trans series representation methods, JKRL outperforms the Trans series models in all indicators. Compared with joint representation models that fuse structure and textual descriptions, JKRL shows the best predictive performance for most relation types. It shows that JKRL not only effectively encodes the representation of the triplet structure, entity description and relationship description text, but also effectively balances the contribution of each vector representation in the learning process through the joint evaluation function, and jointly promotes the representation ability of the KG, thus improving the performance of entity prediction.
• Joint has the best entity prediction performance among all baseline models. JKRL has similar indexes to Joint in N-1, N-N and 1-N relations and is significantly better than Joint in other multiple indexes, which indicates the importance of introducing entity description and relationship description in the JKRL model. Entity description encoding based on CNN and relationship description encoding based on Attention-Bi-LSTM can effectively extract text features and improve the model’s ability of knowledge representation.
• By comparing JKRL and its variant models, we can rank their entity prediction capabilities as follows: JKRL(CNN+A-Bi-LSTM) > JKRL(A-Bi-LSTM) > JKRL(CNN+Bi-LSTM). This shows that the attention mechanism in A-Bi-LSTM plays an extremely important role in encoding the relational description. The attention mechanism calculates the semantic relevance of each word and different relations in the relational description, so as to dynamically select the most relevant information from the relational description and obtain the accurate relational text representation. However, due to the high correlation between the entity description text and the entity itself, the relative distance between the word and the entity has little connection with the expression of the word semantics. If A-Bi-LSTM is used to encode entity description, the attention mechanism has limited improvement in the extraction of semantic features, and the introduction of position features will bring noise and cause the overall model performance to decline. However, CNN can effectively capture local features in the entity description text by sliding on the text sequence through the convolution kernel and has the advantage of parallel computation with higher computational efficiency.

In addition, we visually compared the prediction performance of Joint, Jointly(Bi-LSTM), Jointly(A-Bi-LSTM), JKRL(CNN+Bi-LSTM), JKRL(A-Bi-LSTM) and JKRL(CNN+A-Bi-LSTM) under different relationship types. The specific comparison results are shown in Figure 6, where “PH” and “PT” represent head entity prediction and tail entity prediction, respectively.

2.

Triple classification task

Table 5. Experimental results of triplet classification.

Model	Accuracy
TransE	0.798
TransH	0.877
TransR	0.839
TransD	0.880
DKRL	0.901
Jointly(Bi-LSTM)	0.905
Jointly(A-Bi-LSTM)	0.915
Joint	0.886
JKRL(CNN+Bi-LSTM)	0.918
JKRL(A-Bi-LSTM)	0.926
JKRL(CNN+A-Bi-LSTM)	0.932

shows the experimental results of the triple classification tasks of all models on the FB15K dataset.

Through the observation and analysis of the data in the table, we can draw the following conclusions:

• Compared with all baseline models, JKRL has the best performance, with an accuracy rate of 93.2% on the triplet classification task, which greatly exceeds the structure-based Trans series representation models. Compared with DKRL and Joint, it increased by 3.1% and 4.6%, respectively, and the accuracy was significantly improved. Compared to Jointly, the accuracy was improved by 1.7%. This proves the effectiveness of JKRL, which can effectively extract the semantic features of text description to improve the representation of entities and relations.
• The triple classification accuracy of the JKRL(Bi-LSTM) model is 91.8%, which is 1.4% lower than that of the JKRL(A-Bi-LSTM) model. This shows the effectiveness of the Attention-Bi-LSTM encoding the relational description. The attention mechanism is introduced to calculate the weight of each word, which reduces the noise at the word level in the text description, and can dynamically select the most relevant content from the relationship description for modeling, so as to obtain a more accurate text representation of the relationship.
• JKRL(A-Bi-LSTM) uses A-Bi-LSTM to encode both entity description and relationship description. In the case of increased model complexity, the accuracy rate is slightly decreased, decreasing by 0.6% compared with JKRL(CNN+A-Bi-LSTM). This once again verifies the validity of entity description encoding using CNN, which can not only effectively capture the deep semantic features in the entity description text, but also has higher computational efficiency.

3.

Runtime comparison

Different model architectures have a great impact on the computational workload. We calculated the training time consumption of JKRL and its variant models for triplet classification tasks, as shown in

Table 6. Training time results of JKRL and its variant models on triplet classification.

Model	Training Time	Triple Classification Accuracy
JKRL(CNN+Bi-LSTM)	8.7834 h	0.918
JKRL(A-Bi-LSTM)	25.8745 h	0.926
JKRL(CNN+A-Bi-LSTM)	10.2359 h	0.932

.

As can be seen from the table, JKRL(CNN+A-Bi-LSTM) took a 16.5% longer time than JKRL(CNN+Bi-LSTM) in training. On the basis of Bi-LSTM, the attention mechanism calculates the semantic relevance of each word and different relations in the relationship description. Because the text of the relationship description is short and contains relatively few words, the number of word weights needed to be calculated for each relationship description is relatively small, which makes A-Bi-LSTM effectively improve the representation ability of relational description with a small increase in consumption time. JKRL(CNN+A-Bi-LSTM) saves 60.4% of the time compared to JKRL(A-Bi-LSTM) training. Since the entity description text is long and has a large amount of data, using a CNN to encode it can effectively capture the local features in the entity description text, and has the advantage of parallel computing. Therefore, it not only captures the deep semantic features of the relational description more effectively, but also greatly improves the operation efficiency of the whole model.

5. Conclusions

In view of the fact that most knowledge representation models based on text description cannot fully mine the semantic features of text description, this paper proposes a KG representation method, JKRL, based on structure and text description, which jointly learns the structure-based representation and text-based representation of entities and relationships and effectively balances the contribution of triple structure and text description in the process of vector joint representation learning. The method is based on training the structured representation of entities and relations using TransE and using a convolutional neural network to encode the entity descriptions for obtaining textual representations of entities. Specifically, to extract the semantic features of the hidden text relations in the relationship descriptions, an Attention-Bi-LSTM text encoder is designed to encode the relationship descriptions. Firstly, position features are introduced into the word features, then Bi-LSTM is used to encode the words, and finally, an attention layer is used to compute the weights of each word based on the attention mechanism and dynamically select the most relevant content based on the weights to obtain the textual representation of the relationship. This method performs joint representation learning based on structured and textual representations, effectively balancing the contribution of structured and textual representations in the vector learning process. The experimental results show that the JKRL model proposed in this paper has strong competitiveness compared with the most advanced methods and can improve the quality of the vector representation of entities and relationships in the KG.

The method proposed in this paper starts from the perspective of additional information, introduces text description information and improves the way of modeling information to mine more accurate and richer semantic features to enhance the knowledge representation effect of the model. However, this study did not consider the introduction of multiple sources of information, such as relationship paths and entity types. How to integrate multiple sources of information with triple structures to improve the accuracy of knowledge representation is a future research direction.