A Triple Relation Network for Joint Entity and Relation Extraction

Abstract

Recent methods of extracting relational triples mainly focus on the overlapping problem and achieve considerable performance. Most previous approaches extract triples solely conditioned on context words, but ignore the potential relations among the extracted entities, which will cause incompleteness in succeeding Knowledge Graphs’ (KGs) construction. Since relevant triples give a clue for establishing implicit connections among entities, we propose a Triple Relation Network (Trn) to jointly extract triples, especially handling extracting implicit triples. Specifically, we design an attention-based entity pair encoding module to identify all normal entity pairs directly. To construct implicit connections among these extracted entities in triples, we utilize our triple reasoning module to calculate relevance between two triples. Then, we select the top-K relevant triple pairs and transform them into implicit entity pairs to predict the corresponding implicit relations. We utilize a bipartite matching objective to match normal triples and implicit triples with the corresponding labels. Extensive experiments demonstrate the effectiveness of the proposed method on two public benchmarks, and our proposed model significantly outperforms previous strong baselines.

1. Introduction

Relational triples’ extraction aims to predict the semantic relation between the two entities in an unstructured text, which plays a vital role in succeeding Knowledge Graphs’ (KGs) construction. The relational triples are normally presented in the form of (subject, relation, object). Consider this sentence: “Beijing is known as the capital of China.” From the sentence, an ideal triple extraction would comprise Beijing, Capital_of, China, in which Capital_of is the relation of Beijing and China. Therefore, the relational triple in this example is expressed as (Beijing, Capital_of, China).

Early works in relational triple extraction [1,2] adopted a pipeline manner to extract entities, classified relations in two separate processes, and achieved promising performance. The pipeline methods suffer from the error propagation problem, lacking interaction between the two stages. To address the problem, subsequent works proposed joint learning of entities and relations, which are mainly categorized as feature-based [3,4,5,6] and neural-network-based [7,8]. To tackle extensive manual efforts in feature-based models, recent studies observed neural-based models to perform better in the task. However, neural-based models like [8] fail to extract overlapping triples that the entities share with other triples in a sentence, as Figure 1 illustrates. Many studies, such as sequence-tagging-based [9,10,11,12,13,14], sequence-to-sequence (Seq2Seq)-based [15,16,17,18,19,20], and table-filling-based [21,22,23,24] methods mainly handle the overlapping problem.

However, most existing methods only extract triples via context words that are explicitly expressed in a sentence [25]. In fact, there are many implicit relations between entities. Figure 2 shows an example: the explicit relations Work_for and Locate_in can be easily recognized by recent models via context words played for and in, respectively, but hardly recognize the implicit relation Live_in due to the lack of the explicit expression in the sentence. That means many extracted entities that have certain connections with each other could be ignored. Consequently, such a scenario will cause the incompleteness of KGs’ construction [26,27,28]. To handle these scenarios, the model needs to be equipped with a reasoning ability.

Given the deficiency of the reasoning ability in previous works, we aim to explore more potential information in the raw sentence. The relevant triples provide critical evidence in establishing implicit connections among entities. As shown in Figure 2, the entity pairs (Fan Zhiyi, Shanghai Shenhua) and (Shanghai Shenhua, Shanghai) are overlapped with Shanghai Shenhua, which gives a strong clue to connect with entities FanZhiyi and Shanghai. The two selected entities are combined into a new triple (Fan Zhiyi, Live_in, Shanghai). The implicit relation Live_in can be inferred from the triple pair structures (A, Work_for, B) and (B, Locate_in, C), which means the implicit connections can be learned from the structures. Therefore, we extract the critical evidence from the relevant relational triples.

In this paper, we propose a Triple Relation Network (Trn) to address the implicit relational triples’ extraction problem. Given a sentence, Trn first utilizes the encoder to obtain the token representations. Most joint models first extract possible head entities and then match with all tail entities, leading to high complexity ($N^2$ entity pairs) and performance degradation introduced by unrelated entity pairs. To solve this problem, we design an attention-based entity pair encoding module to obtain the entity pair representations. We obtain the corresponding relation representations for normal entity pairs. A triple reasoning module is introduced to calculate the probability of each triple (concatenation of entity pairs and their relations) pair for reasoning a new triple to find the implicit entity pairs. The higher probability indicates that there is an implicit connection among entities in two triples. Our triple generation module is used to decode the entities in a span-based manner. To calculate the loss of implicit triples and normal triples with the corresponding training labels, we utilize the bipartite matching loss in the training stage. The experimental results show that our proposed Trn outperforms the previous strong methods on two benchmarks and performs better in extracting implicit triples.

Our main contributions include the following:

• We propose a novel Triple Relation Network framework to extract the relational triples, especially implicit triples existing in raw sentences. This implies that the designed model possesses the power of reasoning to search for comprehensive information in the triples’ extraction task.
• We instantiate the proposed Trn, including an attention-based entity pair encoding module to collect normal entity pairs directly and a triple reasoning module to find the implicit connections among all entities. Finally, a bipartite matching training objective is introduced to update the model parameters.
• The main experiments show that the proposed model outperforms the previous strong baselines on two benchmarks, and analytic experiments verify our motivation in potential relations among the extracted entities.

2. Related Work

Relational triples’ extraction [29] is a task in which factual knowledge is mined from texts. It is a well-studied task in information extraction. It is also an important step for the construction of large-scale KGs such as DBpedia [30], Freebase [31], and Knowledge Vault [32]. The task is mainly divided into two manners.

Early studies, such as [1,2], addressed the task in a pipeline manner. They extracted both entities and relations, where entities are extracted first, and then, the relation between the extracted entities is predicted. These methods ignore the interactions between entities and relations and cause the error propagation problem.

To prevent error propagation in the pipeline manner, subsequent studies addressed the task in a joint entity and relation extraction manner, which jointly extracts entities and relations. Traditional joint models are feature-based methods [3,4,5,6], which heavily rely on feature engineering with manual efforts. To eliminate hand-crafted features, recent studies proposed the neural-network-based method, which achieves state-of-the-art performance. However, most joint neural models, such as [7], extract entity pairs and predict the corresponding relations between them through parameter sharing, like the pipeline manner, in which the decoding setting is separated and still leads to an error propagation to a certain extent. Unlike parameter sharing, [8] introduced a unified tagging scheme that transforms the task into an end-to-end sequence tagging problem. Such a joint decoding setting identifies the entities and relations in one shot instead of two separate processes, Named Entity Recognition (NER) and Relation Classification (RC), and naturally handle the error propagation problem.

Most of the previous joint models suffer from the overlapping problem in which triples are overlapped with one or two entities in a sentence. The problem directly affects sequence tagging models like [8], since they only recognize each entity token by one tag. To solve this problem, subsequent studies can be roughly categorized into three frameworks: sequence-tagging-based [9,10,11,12,13,14], sequence-to-sequence-based [15,16,17,18,19,20], and table-filling-based [21,22,23,24]. Sequence tagging models the entities and relations’ extraction task as a tagging problem. To extract an entity (subject or object), the tagging sequences are used to recognize the start position and end position and sometimes determine the relations simultaneously based on the subjects extracted. Sequence-to-sequence models regard the triples as a sequence and view the entities and relations’ extraction task as a generation task. Basically, the method uses an encoder–decoder architecture just like other generation tasks in NLP. Table filling converts the entities and relations’ extraction task into filling an $n\times n$ table problem (n is the number of tokens in a sentence). The table elements usually indicate the start and end positions of two entities or the entity type of each entity.

Despite their success, most methods only capture the connection between two entities by explicit context words in a sentence. Unfortunately, it is hard for them to identify relational facts, which are not explicitly expressed in a sentence [25], leading to the underutilization of resources and the incompleteness of KGs’ construction [26,27]. To capture the inner connections, [10] proposed a method named TME to use entity embedding translation to predict relations. However, TME needs a complicated tagging scheme (position, type, relation) and cannot handle symmetric relations.

To overcome the aforementioned problems, we designed a framework called Trn. Following the Seq2Seq-based methods, we modeled triples as a sequence and naturally dealt with the triple overlapping problem. We propose a triple reasoning module to construct implicit connections among these entity pairs. It can find the two relevant triples to infer a new triple and obtain these two implicit entities inside them.

3. Proposed Framework

In this section, we introduce the proposed model Trn. The main architecture of Trn is shown in Figure 3. Section 3.1 introduces our sentence encoder. Section 3.2 is our entity pair encoding module. Section 3.3 is our triple reasoning module used to find implicit entity pairs. Section 3.4 shows our triple generation module. Section 3.5 describes the details of the bipartite matching training objective.

3.1. Sentence Encoder

Let $S=\left[{\mathit{w}}_{\mathbf{1}},{\mathit{w}}_{\mathbf{2}},\cdots ,{\mathit{w}}_{\mathbf{m}}\right]$ be an input sentence, where ${\mathit{w}}_{\mathbf{i}}$$(1\le i\le m)$ is the i-th token in the sentence and m is the sentence length. We utilized the BERT [33] model as the encoder. The output of the encoder is denoted as follows:

$$H=\left\{h_1,h_2,\cdots ,h_m\right\}$$

(1)

where $h_i$ $(1\le i\le m)\in {\mathbb{R}}^d$ is the encoding representation of the i-th token in a sentence (including start symbol [CLS] and end symbol [SEP]) and d is the hidden size.

3.2. Entity Pair Encoding

After we obtained the token representations of the current sentence from the sentence encoder, we randomly initialized n-sized raw entity pairs as the initial ones, which are explicitly recognized by context words. Similar to the decoder of [34], we used the self-attention and cross-attention parts as the entity pair encoding module. Through the self-attention layer, we can capture the dependency information regardless of the distance between entity pairs. Through the cross-attention layer, we can capture contextual information to establish an entity pair representation. Since there is no causal mask in the self-attention layer, we fused the information into raw entity pairs in parallel. Thus, we set n to a relatively large number to ensure we can extract enough entity pairs in a sentence. Consequently, the representations of n entity pairs $E=\left\{e_1,e_2,\dots ,e_n\mid e_i\in {\mathbb{R}}^d\right\}$ is computed as:

$$E=EntityPairEncoding\left(H;E_{raw}\right)$$

(2)

where $E_{raw}$ denotes n raw entity pairs.

We utilized multi-head attention [35] as the entity pair encoding module. The details of self-attention and cross-attention are described as:

$$Attention(Q,K,V)=softmax\left(\frac{Q{K}^T}{\sqrt{d}}\right)V$$

(3)

where Q, K, V are three matrices that are used to transform embeddings to query, key, and value. $\sqrt{d}$ is the scaling factor used to control the dot product value. We set $Q=K=V=E_{raw}$ for the self-attention layer and $K=V=H$ for the cross-attention layer.

The multi-head attention is defined by:

$$hea{d}_i=Attention(Q{W}_i^q,K{W}_i^k,V{W}_i^v)$$

(4)

where $W_i^q$, $W_i^k$, $W_i^v$ are learnable matrices and $hea{d}_i$ is the i-th head representation.

Finally, we obtain the entity pair representations:

$$\mathbb{E}=Concat(hea{d}_1,\dots ,hea{d}_o)W^o$$

(5)

where o is the number of heads in the attention mechanism and $W^o$ is the learnable matrix.

3.3. Triple Reasoning

The triple reasoning module aims to find the implicit entity pairs among relational triples. Due to the lack of context words to recognize the connection between two implicit entities, we can only rely on sufficient information provided by the extracted triples, e.g., in Figure 2, (Fan Zhiyi, Live_in, Shanghai) is reasoned by (Fan Zhiyi, Work_for, Shanghai Shenhua) and (Shanghai Shenhua, Locate_in, Shanghai). We also considered a single triple reasoning case, e.g., from the triple (Beijing, Capital_of, China), we can obtain implicit triple (Beijing, Locate_in, China) or (China, Contain, Beijing), which infers the new relation Locate_in or Contain from the two same entities. We treated the single triple reasoning also as the triple pair reasoning by simply utilizing the triple and its duplicate as the input.

Basically, we utilized the triples to reason the implicit entity pairs in three steps. First, for each entity pair representation, we collected the relation representation between them to form the triple representation. Then, we calculated the relevant scores with all other triples (including itself) to obtain the probabilities for reasoning a new triple. Last, we used the top-K strategy to select a fixed number of implicit entity pairs.

Formally, given the current entity pair $e_i$ and corresponding relation type $r_i\in {\mathbb{R}}^d$, the probabilities of k implicit entity pairs are defined as:

$$P\left(E_{imp}\right)=\underset{i,j\in \{1,2,\dots ,n\}}{argmax}\prod ^{k}p\left(e_{ij}^{imp}\mid e_i\parallel r_i,e_j\parallel r_j;\theta \right)$$

(6)

where $E_{imp}$ is the entity pair set, k is the number of implicit triples that we set from our strategy described in the top-K strategy, and $e_{ij}^{imp}\in {\mathbb{R}}^d$ is the implicit entity pair in which i and j are the indices of two selected triples. The entity pair-embeddings- and relation-embeddings-based feature vectors are concatenated with $\parallel$ to obtain the triples’ representation.

Triple Collection: To infer the potential relationship between two entities, we need to capture the information from the triples that we predicted through explicit expressions. In this part, we calculate the relation representation $r_i$ for each entity pair $e_i$:

$$r_i=FFN\left(e_i\right)$$

(7)

where $FFN$ is a feed-forward network. Then, we obtain the triple representation $t_i=(e_i\parallel r_i)\in {\mathbb{R}}^{2d}$ as the input of the left upper part of Figure 3.

Scoring Pool: We computed the probability to decide whether an implicit entity pair could be constructed via a triple pair. Specifically, given the input set of triples $T=\left[t_1,t_2,\cdots ,t_n\right]$, for any triple $t_i$ coupled with $t_j$, the score ${\omega }^{ij}$ is computed as:

$${\omega }^{ij}=Relu(W^s·(t_i\parallel t_j)+b^s),\left(i,j=1,2,\dots ,n\right)$$

(8)

where $W^s\in {\mathbb{R}}^{4d\times 1}$ and $b^s$ are learnable parameters. Figure 4 corresponds to Equation (8). Note that since not all triple pairs are useful for predicting the new one, the triple that is irrelevant with respect to the others is supposed to be set at lower scores. Therefore, we did not use the attention mechanism [36] to calculate the probabilities with others because all weight scores were restricted to sum to 1. The higher the weight score, the higher the probability of the triple pair inferring the implicit triple is. Finally, we formed a scoring pool that collects all the scores calculated in a total of n triples with each other.

Top-K Strategy: We introduced our top-K strategy. We selected the top-k relevant triple pairs based on the scoring pool as candidates, which were used for inferring implicit triples. When $k=1$, our strategy is downgraded to be a “max” approach in selecting the best triple pair to conduct reasoning. The output is a weighted sum of the features of implicit entity pair $e_{ij}^{imp}$ inferred from triple pair $t_i$ and $t_j$. Let $\mathcal{K}$ denote the indices of the best k triple pairs. The implicit entity pair is computed as:

$$e_{ij}^{imp}=W^{\psi }·{\omega }^{ij}(t_i\parallel t_j),\forall (i,j)\in \mathcal{K}$$

(9)

where $W^{\psi }\in {\mathbb{R}}^{4d\times d}$. In this paper, we set $k=5$ as the default.

3.4. Triple Generation

Through the triple reasoning module, we obtained k implicit entity pairs and added them into E. All entities are recognized from the start positions to the end positions in the sentence with m tokens. We decided the positions of two entities that gave the max values of four pointer probabilities to form the best entity pair combination.

Given an entity pair $e_i$ from E, the classification can be obtained by:

$$p^r=ML{P}_r\left(e_i\right)$$

(10)

$$H_{fuse}=Dup(e_i,m)+W^{x,y}H$$

(11)

$$p^{x,y}=ML{P}_{x,y}\left(H_{fuse}\right)$$

(12)

where $p^r$ denotes the probability of the relation type, $p^{x,y}$ is the probabilities of starting or ending indices for an entity pair, and superscripts $x,y$ are the combinations of $head/tail$ and $start/end$, i.e., $p^{head,start}$, $p^{head,end}$, $p^{tail,start}$, $p^{tail,end}$. $MLP$ denotes multi-layer perceptron (same as the $FFN$ of Equation (7)) with the softmax function. $W^{x,y}$ (superscript combinations are the same as $p^{x,y}$) are learnable parameters. $Dup(·)$ will duplicate $e_i$m times into shape ${\mathbb{R}}^{m\times d}$. Therefore, each predicted triple in the set is denoted as ${\widehat{y}}_i=(p_i^r,p_i^{head,start},p_i^{head,end},p_i^{tail,start},p_i^{tail,end})$, and each golden triple in our training labels is preprocessed as the same structure $y_i=(r_i,hea{d}_i^{start},hea{d}_i^{end},tai{l}_i^{start},tai{l}_i^{end})$.

3.5. Training Objective

Since our training labels are disordered, the main difficulty in training is that our predicted triples must be in line with the order of the golden triples. Unfortunately, there are no annotations of implicit triples in either dataset, so it is impossible to be certain that the predicted implicit triples exactly correspond to the position of the golden implicit triples. Besides, the model extremely focuses on predicting the order of multiple triples instead of learning how to generate accurate triples, which hurts the performance in the training stage. To cope with these permutation problems, we need to find an optimal matching strategy between the golden triple set and the predicted triple set before computing the training objective. We used a set prediction loss, which can produce an optimal bipartite matching by using the Hungarian algorithm [37]. To obtain an optimal matching, we searched for a permutation of n elements $\widehat{\alpha }\in \mathsf{\Omega }(n+k)$ with the lowest cost as the first step:

$$\widehat{\alpha }=\underset{\alpha \in \mathsf{\Omega }(n+k)}{argmin}\sum _i^{n+k}{\mathcal{L}}_{match}(y_i,{\widehat{y}}_{\alpha \left(i\right)})$$

(13)

where ${\mathcal{L}}_{match}(y_i,{\widehat{y}}_{\alpha \left(i\right)})$ is a triple pair matching cost, $y_i$ is a golden triple, and $y_{\alpha \left(i\right)}$ is a prediction with index $\alpha \left(i\right)$. $\mathsf{\Omega }(n+k)$ is the space of all permutations of an $n+k$-sized set. ${\mathcal{L}}_{match}(y_i,{\widehat{y}}_{\alpha \left(i\right)})$ is defined as:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{match}}\left(y_i,{\widehat{y}}_{\alpha \left(i\right)}\right)& =-1_{\left\{r_i\ne ⌀\right\}}\left[p_{\alpha \left(i\right)}^r\left(r_i\right)\right.\hfill \\ \hfill & +p_{\alpha \left(i\right)}^{\mathrm{head},\mathrm{start}}\left(\mathrm{head}{}_i^{\mathrm{start}}\right)\hfill \\ \hfill & +p_{\alpha \left(i\right)}^{\mathrm{head},\mathrm{end}}\left(\mathrm{head}{}_i^{\mathrm{end}}\right)\hfill \\ \hfill & +p_{\alpha \left(i\right)}^{\mathrm{tail},\mathrm{start}}\left(\mathrm{tail}{}_i^{\mathrm{start}}\right)\hfill \\ \hfill & \left.+p_{\alpha \left(i\right)}^{\mathrm{tail},\mathrm{end}}\left(\mathrm{tail}{}_i^{\mathrm{end}}\right)\right]\hfill \end{array}$$

(14)

Then, we obtain the optimal matching $\widehat{\alpha }$ between the permutation of golden triples and predictions. We define the final loss ${\mathcal{L}}_{loss}$ as:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{loss}}(y,\widehat{y})& =\sum _{i=1}^{n+k}\left\{-log{p}_{\widehat{\alpha }\left(i\right)}^r\left(r_i\right)\right.\hfill \\ \hfill & +1_{\left\{r_i\ne ⌀\right\}}\left[-log{p}_{\widehat{\alpha }\left(i\right)}^{\mathrm{head},\mathrm{start}}\left(\mathrm{head}{}_i^{\mathrm{start}}\right)\right.\hfill \\ \hfill & -log{p}_{\widehat{\alpha }\left(i\right)}^{\mathrm{head},\mathrm{end}}\left(\mathrm{head}{}_i^{\mathrm{end}}\right)\hfill \\ \hfill & -log{p}_{\widehat{\alpha }\left(i\right)}^{\mathrm{tail},\mathrm{start}}\left(\mathrm{tail}{}_i^{\mathrm{start}}\right)\hfill \\ \hfill & \left.\left.-log{p}_{\widehat{\alpha }\left(i\right)}^{\mathrm{tail},\mathrm{end}}\left(\mathrm{tail}{}_i^{\mathrm{end}}\right)\right]\right\}\hfill \end{array}$$

(15)

where $\widehat{\alpha }\left(i\right)$ is the element of the optimal permutation.

4. Experiments and Discussion

4.1. Experimental Setup

Datasets In the experimental setup, we evaluated our proposed model on the two widely used public datasets: New York Times (NYT) [38] and WebNLG [39].

NYT collects sentences from the New York Times corpus, and it contains 24 predefined relation types. Since there are many versions of the NYT dataset, we adopted the preprocessed version used in [15].

WebNLG contains 171 (The number is miswritten by many works. We followed the correct number from [23].) predefined relation types. The dataset was originally created for the Natural Language Generation (NLG) task.

We split the sentences into three categories [15] according to different overlapping patterns of triples: Normal, SEO, and EPO. The statistics of the two datasets are shown in

Table 1. Statistics of NYT and WebNLG.

Dataset	NYT		WebNLG
	Train	Test	Train	Test
Normal	37,013	3266	1596	246
SEO	14,735	1297	3406	457
EPO	9782	978	227	26
ALL	56,195	5000	5019	703

.

Evaluation We utilized the standard micro-F1-scores to evaluate the proposed model on both NYT and WebNLG. The predicted triple is regarded as correct if the relation type and two entities are all correct. Note that the entity is correct if the last words of the entity are correct because both datasets only annotated the last word of every entity. Therefore, we only performed partial matching evaluations.

Implementation Details We implemented our Trn with the Pytorch framework. For a fair comparison with previous works [12,23], we used BERT${}_{base\_cased}$ in our experiments as the encoder and set the max length of a sentence to 100. The learning rate of BERT was set to ${10}^{-5}$, and the attention layers and triple reasoning module were all set to $2\times {10}^{-5}$. The number of entity pairs n was set to 10, and the number of implicit entity pairs k was set to 5. We used Byte Pair Encoding [40] to segment the input sentences. The hidden dimension d was set to 768. We set 3 layers in the entity pair encoding module. The dropout [41] of the attention layer was set to 0.1. We used AdamW [42] to update Trn to minimize the loss function defined in Section 3.5 and set the weight decay as 0.02. We trained the designed model for 200 epochs with the batch size as 32/16 on NYT/WebNLG.

4.2. Comparative Models

We compared Trn with several previous models:

• NovelTagging [8] proposes a novel tagging scheme to transform the task into a sequence labeling problem.
• CopyRE [15] proposes an end-to-end based on sequence-to-sequence learning with a copy mechanism, which jointly extracts triples as the sequence form.
• CopyRE${}_{RL}$ [16] considers the order of the relational facts extracted from the sentence to be also important. They utilized reinforcement learning (RL) in the sequence-to-sequence model to learn the extraction order.
• TME [10] proposes a joint model that is capable of discovering multiple triples in a sentence via ranking with a translation mechanism. They used a tri-part tagging scheme (position, type, entity) to obtain entity features.
• ETL-Span [11] proposes a decomposition-based tagging scheme. They decomposed the joint extraction task into HE extraction and TER extraction, which are interrelated subtasks. Their model can not only capture the semantic interdependency between different steps, but also reduces noise from irrelevant entity pairs.
• CopyMTL [17] proposes a Multi-Task Learning (MTL) framework equipped with a copy mechanism to predict the multi-token entities and accurately differentiate head and tail entities.
• WDec [18] proposes two approaches to use the sequence-to-sequence architecture to jointly extract entities and relations.
• CasRel [12] proposes a novel cascade binary tagging framework. They treated relations as functions that map subjects to objects in a sentence to solve the overlapping problem.
• PMEI [43] designs a multitask learning architecture to extract relevant features that correlate with entity recognition and relation extraction from the input. They controlled the injection of early predictions to ensure that the extracted task-specific representations were good for classification.
• TPLinker [23] proposes a novel handshaking tagging scheme to perform the one-stage extraction. TPLinker is capable of discovering overlapping relations sharing one or both entities to address the exposure bias problem.
• RMAN [14] proposed a multi-feature fusion sentence representation and decoder sequence annotation to deal with overlapping triples.
• CGT${}_{UniLM}$ [20] proposes a model, contrastive triple extraction with a generative transformer. They introduced a single shared transformer module for encoder–decoder-based generation.
• RIFRE [13] designs a tagging framework, which first extracts all the candidate head entities and then labels the tail entities with corresponding relations.

4.3. Performance

4.3.1. Main Results

From

Table 2. The performance of our proposed Trn and previous models on the NYT and WebNLG test sets. Trn−TRm indicates the model does not use the triples’ reasoning module presented in Section 3.3. The best scores are in bold font.

Models	NYT			WebNLG
	Precision	Recall	F1	Precision	Recall	F1
NovelTagging [8]	62.4	31.7	42.0	52.5	19.3	28.3
CopyRE [15]	61.0	56.6	58.7	37.7	36.4	37.1
CopyRE${}_{RL}$ [16]	77.9	67.2	72.1	63.3	59.9	61.6
TME [10]	69.6	47.8	56.7	-	-	-
ETL-Span [11]	84.9	72.3	78.1	84.0	91.5	87.6
CopyMTL [17]	75.7	68.7	72.0	58.0	54.9	56.4
WDec [18]	94.5	76.2	84.4	-	-	-
CasRel [12]	89.7	89.5	89.6	93.4	90.1	91.8
PMEI [43]	90.5	89.8	90.1	91.0	92.9	92.0
TPLinker [23]	91.3	92.5	91.9	91.8	92.0	91.9
RMAN [14]	87.1	83.8	85.4	83.6	85.3	84.5
CGT [20]	94.7	84.2	89.1	92.9	75.6	83.4
RIFRE [13]	93.6	90.5	92.0	93.3	92.0	92.6
Trn (Our method)	93.0	92.3	92.6	93.5	92.7	93.1
Trn−TRm	92.4	92.1	92.2	93.4	92.4	92.8

, we can observe the results of these baseline models and Trn: our proposed model Trn significantly outperforms all previous baseline models on NYT and WebNLG. Concretely, the proposed model achieves $0.6\%$ and $0.5\%$ absolute F1-score improvements over the previous method RIFRE [13] on NYT and WebNLG, respectively. Trn−TRm outperforms all the baseline models even without using the triple reasoning module. This demonstrates that our basic architecture is more effective in the relational triples’ extraction task. In addition, for the model Trn−TRm without using the triple reasoning module, we can observe that there are $0.4\%$ and $0.3\%$ F1-score improvementscompared with Trn on NYT and WebNLG, respectively. This proves the triple reasoning module can truly extract additional relational triples.

Compared to TME [10], capturing inner connections, Trn performs better on NYT. This demonstrates that Trn has a strong capability of identifying implicit triples.

We can also observe from Table 2 that there is a gap between the performance on NYT and WebNLG for most baseline models. In detail, from Table 1, we can find that the ratio of overlapping triples in WebNLG is larger than the ratio in NYT. The imbalanced data distribution leads to worse performance on WebNLG since most baselines are weak at dealing with overlapping triples. We also find that [23] obtains a fair performance on the two benchmarks, which benefits from their handshaking architecture.

4.3.2. Detailed Results on Different Types of Sentences

We compared the ability of the models in extracting relational facts from sentences with different numbers of triples and different overlapping patterns. The detailed results are shown in

Table 3. The F1-sores on sentences with different numbers of triples. N is the number of triples in a sentence. The best scores are in bold, while the second-best scores are underlined.

Models	NYT								WebNLG
	Normal	SEO	EPO	N = 1	N = 2	N = 3	N = 4	N ≥ 5	Normal	SEO	EPO	N = 1	N = 2	N = 3	N = 4	N ≥ 5
CopyRE	66.0	48.6	55.0	67.1	58.6	52.0	53.6	30.0	59.2	33.0	36.6	59.2	42.5	31.7	24.2	30.0
CopyRE${}_{RL}$	71.2	69.4	72.8	71.7	72.6	72.5	77.9	45.9	65.4	60.1	67.4	63.4	62.2	64.4	57.2	55.7
CasRel	87.3	91.4	92.0	88.2	90.3	91.9	94.2	83.7	89.4	92.2	94.7	89.3	90.8	94.2	92.4	90.9
TPLinker	90.1	93.4	94.0	90.0	92.9	93.1	96.1	90.0	87.9	92.5	95.3	88.0	90.1	94.6	93.3	91.6
RIFRE	90.7	93.2	93.5	90.7	92.8	93.4	94.8	89.6	90.1	93.1	94.7	90.2	92.0	94.8	93.0	92.0
Trn	90.5	94.6	94.6	90.6	93.5	94.1	96.4	91.9	90.4	93.6	90.3	90.0	91.7	94.5	94.8	94.0

. The results indicate that the designed model can not only extract multiple relational triples, but also handles the overlapping problem, which is popular in this task.

From Table 3, we divided the sentences into eight sub-classes for each test set. We observed that the designed model achieves 10 best scores and 4 second-best scores on the different numbers of triples over the two benchmarks. Our proposed model achieves $1.9\%$ and $2.0\%$ improvements on NYT and WebNLG over the second-best scores when $N\ge 5$. The results indicate that the designed model is more effective in handling difficult scenarios.

From the overlapping patterns perspective, we divided the sentences into three sub-classes (Normal, SEO, EPO). Our proposed model Trn also achieves the best scores in most cases compared to the previous baselines. There are $1.2\%$ and $0.5\%$ F1-score improvements compared with the previous best scores in SEO sentences on NYT and WebNLG, respectively, and $0.6\%$ in EPO sentences on NYT. The results indicate that the proposed model is more suitable for handling complicated scenarios. We found that the implicit triples always overlapped with others in a sentence. Therefore, we can infer that the improvements of extracting overlapping triples demonstrate that the designed model can truly extract more implicit triples.

We found that the EPO performance of Trn on WebNLG was worse than TPLinker from

Table 4. The EPO performance of Trn and TPLinker on the WebNLG test set.

Model	Precision	Recall	F1
TPLinker	97.6	94.3	96.0
Trn	95.5	85.7	90.3

. To explore the reasons, we conducted the detailed experiment only on EPO sentences. The gap of the recall indicates that the proposed model cannot reason more complete triples than TPLinker. We speculated that one reason is that Trn needs much more EPO data to fit and evaluate. We found that all previous work classified overlapping pattern A→B and B→A (A, B are two different entities; → is the relation between them) into SEO rather than the EPO defined by [15], which means implicit triples mostly come from SEO sentences. Thus, EPO sentences only contain A→B and A→B, for which we can only reason implicit triples in a multi-labeled pattern. The designed model mainly reasons the implicit triples via SEO sentences. Therefore, the single triple reasoning, i.e., A→B reasons multi-labeled A→B categorized as EPO very few times.

4.4. Ablation Study

Results on Different Model Settings: We conducted ablation studies to prove the inferential effectiveness on our subset (We collected triple combinations in a sentence, such as A→B, B→C, coupled with A→C or A→B and A→C, coupled with B→C or A→B, coupled with B→A combinations. A, B and C are three different entities.), which were manually selected from NYT. We selected the 100 overlapping pattern sentences, which are rich in implicit triples. Then, we conducted our experiments on the previous models and ours with different ablation settings:

• Trn−TRm removes the whole triple reasoning module.
• Trn−TRm-Seq removes the triple reasoning module, adds a causal mask to the entity pair encoding module to predict the triples order by order (does not distinguish the difference between normal triples and implicit triples), and uses NLL loss as the training objective.

We compared the performance of different settings on our subset. The results are illustrated in

Table 5. Performance of different models on our subset.

Model	F1 Score
CGT	79.1
TPLinker	84.9
RIFRE	85.3
Trn−TRm-Seq	82.2
Trn−TRm	87.6
Trn	91.4

. Compared with these different models, we can observe that: (1) Trn brings a significant improvement over Trn−TRm on the subset. This indicates that our triple reasoning module can capture the information among triples and construct the implicit connections between entities. (2) Trn−TRm also brings improvements over Trn−TRm-Seq. We considered that it benefits from the training strategy. The bipartite training strategy successfully matches the normal triples and implicit triples with the ground truth at the training stage. (3) Trn−TRm-Seq is close to TPLinker and RIFRE on our subset. Note that the sequence-to-sequence architecture brings improvements over the generative model CGT. We considered that it benefits from our entity pair encoding module. (4) Trn−TRm also outperforms the previous models. Even without using the triple reasoning module, the proposed model obtains a competitive result with respect to the previous methods. This indicates that using our basic structure is effective.

Parameter Analysis: We trained different models and evaluated them on our subset to analyze how the k number of implicit triples affects model performance. From Figure 5, we can observe that when $k=0$ and $n\le 4$, the F1-score presents a tendency to increase. This indicates that the number of normal triples is still insufficient to extract from a raw sentence. There is no obvious increase when $n\ge 4$, which means our basic architecture cannot extract more triples. When $k=5$, we can observe that the F1-scores increase slightly as the number of normal triples progressively increases. This means more implicit triples will be extracted as normal triples increase. It seems the F1-score will converge when $n\ge 5$.

4.5. Case Study

Figure 6 shows the three examples of Trn and TPLinker on our subset. The first example shows an implicit connection (Braga, country, Portugal) by the reversed relation pattern (Portugal, contains, Braga). In the second example, we infer (Edmund, lived, Louisiana) by the multi-hop relation patterns (Edmund, lived, Crowley) and (Louisiana, contains, Crowley). The third example shows (China, contains, Jinghong) in transitive relation patterns (China, contains, Yunnan) and (Yunnan, contains, Jinghong). We also observe that the proposed model can recognize new relations from the orange arrows, although the connections do not givegolden triples.

4.6. Error Analysis

To validate the effectiveness of Trn in handling implicit triples, we conducted a supplemental experiment to explore the factors that affect our proposed model. From

Table 6. Results on triple elements. The triple (E1, R, E2 ) is divided into (E1, E2) and R, where E1 represents the head entity, E2 represents the tail entity, and R represents the relation between E1, E2.

Dataset	Element	EPO			SEO			Overall
		Pre.	Rec.	F1	Pre.	Rec.	F1	Pre.	Rec.	F1
	(E1, E2)	96.3	93.3	94.8	97.1	92.5	94.7	93.7	93.0	93.4
NYT	R	98.4	95.4	96.9	99.4	94.8	97.0	96.9	96.2	96.6
	(E1, R, E2)	96.1	93.1	94.6	96.9	92.4	94.6	93.0	92.3	92.6
	(E1, E2)	97.7	87.8	92.5	96.4	94.9	95.7	95.8	94.9	95.3
WebNLG	R	96.6	86.7	91.4	96.9	95.5	96.2	96.1	95.2	95.6
	(E1, R, E2)	95.5	85.7	90.3	94.3	92.9	93.6	93.5	92.7	93.1

, we observe that there is a trivial gap between ($E1,E2$) and ($E1,R,E2$) for NYT both on EPO and SEO sentences. This implies identifying entity pairs is the bottleneck for Trn. Furthermore, the recall score is relatively lower than the precision, which is the main factor affecting F1-score both on EPO and SEO. This reveals that Trn needs to find more entities that may have connections with each other even if we have already discovered more entity pairs. For WebNLG, we observe that identifying relations comprise the bottleneck on EPO sentences, while entity pairs are for SEO sentences. Note that there is a huge gap between the precision and recall of both ($E1,E2$) and R on EPO sentences. We discussed the reason why the designed model performed worse in Section 4.3. We also found that there is an obvious gap between ($E1,E2$) and ($E1,R,E2$) on SEO sentences and R and ($E1,R,E2$) on EPO sentences. This demonstrates that there is a mismatch between entity pairs and relations. Compared with NYT, we speculated the difference to be that the number of relations in WebNLG is larger than NYT. Therefore, identifying triples is harder for Trn on WebNLG than NYT. To further explore the factors affecting Trn, we classified several error sentences and conducted an analysis. From

Table 7. Error analysis on different cases.

Instances
Sentence #1: He said he had brought in one bomber called Imran from Pakistan’s
North - West Frontier Province, who blew himself up on the road near the
Kandahar airport.
Gold: (Pakistan, administrative_divisions, Province), (Province, country, Pakistan),
(Pakistan, contains, Province)
Pred: (Pakistan, administrative_divisions, Province), (Province, country, Pakistan),
(Pakistan, contains, Province), (Pakistan, contains, Kandahar)
Sentence #2: The vision for this sort of thing has existed in China for a very long
time,” said Wu Yongyi, deputy dean of the International College of Chinese Studies
at East China Normal University in Shanghai, who has been involved in overseas
language instruction missions since the 1980’s.
Gold: (Shanghai, country, China), (China, contains, Shanghai),
(China, administrative_divisions, Shanghai), (Shanghai, contains, University)
Pred: (Shanghai, country, China), (China, contains, Shanghai),
(China, administrative_divisions, Shanghai), (Shanghai, no_relation, University)
Sentence #3: Jeffrey R. Immelt, chairman and chief executive of General Electric,
bounced a $2000 check to the failed New York gubernatorial campaign of William
F. Weld, according to a campaign finance filing released Monday.
Gold: (Weld, place_lived, York), (Immelt, company, Electric),
(Immelt, major_shareholder_of, Electric), (Electric, major_shareholders, Immelt)
Pred: (Immelt, place_lived, York), (Immelt, company, Electric),
(Immelt, major_shareholder_of, Electric), (Electric, major_shareholders, Immelt)
Sentence #4: It is also adding flights on existing routes to several cities, including
Kiev and Odessa in Ukraine, and Dubrovnik and Split in Croatia.
Gold: (Kiev, country, Ukraine), (Ukraine, capital, Kiev),
(Ukraine, administrative_divisions, Kiev), (Croatia, contains, Dubrovnik),
(Ukraine, contains, Odessa), (Ukraine, contains, Kiev)
Pred: (Kiev, country, Ukraine), (Ukraine, capital, Kiev),
(Ukraine, administrative_divisions, Kiev), (Croatia, contains, Dubrovnik),
(Ukraine, contains, Odessa), (Croatia, contains, Split), (Ukraine, contains, Kiev)

:

Wrong Arguments: As Sentence #1 shows, the model incorrectly recognizes an entity mention “Kandahar” as the tail, which is not annotated in the sentence. We argue that there is no evident clue to demonstrate the relation between “Pakistan” and “Kandahar” in context. The wrong prediction will be prohibited if the model is equipped with common sense or real-world knowledge (e.g., Kandahar is a city in Afghanistan; Pakistan is a country near Afghanistan).

No Relation: There is an evident clue in Sentence #2 to declare that “East China Normal University” is in “Shanghai”, but the model incorrectly predicts “no_relation” in our case. We argue that the model normally focuses more on implicit triple reasoning if there are more overlapping pattern triples in a sentence. Thus, it neglects the extraction of normal triples. In particular, such “no_relation” wrong cases are rich in predictions. We can also find in Table 6 that the recall score is much lower than the precision score in any type of sentence.

Arguments Mismatch: We can observe from Sentence #3 that both entities “Immelt” and “York” are annotated in the sentence, but they are not related. We argue that the reason is the same as “no_relation”. It seems balancing the extraction of implicit triples and normal triples is a problem for the proposed model.

No Label: The designed model in Sentence #4 generated the correct triple, but it did not exist in the ground truth label. Generally, this situation also occurs, and it is treated as a wrong case.

5. Conclusions

In this paper, we proposed a novel framework called Trn to jointly extract triples, especially implicit triples. Compared with previous methods, our method is capable of reasoning the implicit triples even if context words are not explicitly expressed in the relation between two entities. Our model will extract more triples that are overlapped with others from the sentence, and we utilized an attention-based entity pair encoding module to extract entity pairs directly. To establish implicit connections among these extracted entities in triples, we designed a triple reasoning module to select relevant triple pairs. We matched the training labels with the corresponding triples using the bipartite method to optimize the matching training objective. Experiment results on two benchmarks validated the effectiveness of our method over different scenarios. Our proposed model also outperformed the most advanced baseline, especially for implicit triples’ extraction.