An Entity Linking Algorithm Derived from Graph Convolutional Network and Contextualized Semantic Relevance

Abstract

In the era of big data, a large amount of unstructured text data springs up every day. Entity linking involves relating the mentions found in the texts to the corresponding entities, which stand for objective things in the real world, in a knowledge base. This task can help computers understand semantics in the texts correctly. Although there have been numerous approaches employed in research such as this, some challenges are still unresolved. Most current approaches utilize neural models to learn important features of the entity and mention context. However, the topic coherence among the referred entities is frequently ignored, which leads to a clear preference for popular entities but poor accuracy for less popular ones. Moreover, the graph-based models face much noise information and high computational complexity. To solve the problems above, the paper puts forward an entity linking algorithm derived from the asymmetric graph convolutional network and the contextualized semantic relevance, which can make full use of the neighboring node information as well as deal with unnecessary noise in the graph. The semantic vector of the candidate entity is obtained by continuously iterating and aggregating the information from neighboring nodes. The contextualized relevance model is a symmetrical structure that is designed to realize the deep semantic measurement between the mentions and the entities. The experimental results show that the proposed algorithm can fully explore the topology information of the graph and dramatically improve the effect of entity linking compared with the baselines.

1. Introduction

It is common that tremendous volumes of textual data spring up on the Web each day. The diverse sources of the data and its large-scale and non-standard presentations pose great challenges for people to collaborate for useful knowledge. Therefore, people urgently hope that computers can understand natural language and process data intelligently to provide knowledge services. In order to describe various entities or concepts existing in the real world, people have begun collaborating to construct knowledge bases (KB), which contain multiple types of the entities, such as organizations, persons, locations, games, movies, positions, etc. Here, Wikipedia [1] and Freebase [2] have drawn wide attention. A Wikipedia page represents an entity, which contains a set of carefully defined relations and attributes. Entity linking (EL) connects the mentions in a text with their corresponding entities in a KB for a better understanding of the text corpus. Figure 1 shows an illustration for the EL task. The plain text “Singapore premiums for Australian kilo bars were quoted unchanged at between 25 45 cents an ounce over spot loco London prices with South Korean origin premiums also steady at 10 20 cents an ounce” has four mentions, which are “Sigapore”, “Australian”, “London”, and “South Korean”. In summary, the mention is ambiguous, which leads to link it to more than one entity in KB. To deal with the problems, the methods used in the previous research mainly focused on designing hand-crafted features to measure the similarity between the mention and the entity in different aspects, including prior popularity [3], context similarity [4], type similarity [5], etc.

Generally, the mentions in the same document are semantically related to each other, and the results of the linking are interactive and dependent. Recently, researchers have tried to construct an entity graph based on KB, and some strategies are used for joint reasoning. Therefore, the EL problem can be cast as an entity ranking problem, and the entity with the highest score is predicted as the correct match. The PageRank and Random Walk algorithm [6,7,8] usually tend to capture topic consistency between the mentions in the text. With ABACO [9], a sub-graph is extracted from a KB and pruned based on nodes’ degree of centrality. Specifically, the algorithm fully considers the Wikipedia page of an entity to compute its semantic similarity with the document topics. The authors in [10] propose to leverage an asymmetric graph convolutional network for entity embeddings, which can integrate global semantic information and latent relation between the entities. KGEL [11] utilizes a knowledge graph to improve the correlation information between the entities. Although traditional graph-based models have achieved significant improvements, they suffer from high computation costs with the increase in the number of candidate entities. In addition, the semantic features implied in context are often ignored due to the data sparsity issue. As the context is composed of words, a large number of studies utilize word embeddings to learn context features [12,13]. For these methods, words are represented by low dimensional vectors in a continuous space, and features for mentions as well as candidate entities are automatically learned from data. They cannot model the topic coherence among various mentions in the text. This paper puts forward the entity linking algorithm derived from asymmetric graph convolutional network and the contextualized semantic relevance (GCNCS), which can utilize neighboring node information and alleviate the problem of excessive noise.

GCNCS differs from the state of the art in how it captures the topic coherence among the various mentions in the entire document. It depends on the construction of the entity graph. For this purpose, the model adopts a distilled version of bidirectional encoder representation from transformers (DistilBERT) to create an asymmetric graph whose nodes are entities and edges. The asymmetric graph is a crucial feature for measuring the distance between the candidate entities. Therefore, our method explores more possibilities between the candidate entities and avoids isolated nodes. The asymmetric graph convolutional network aggregates neighborhoods’ features across related entities in a graph by the flexible encoding of entity graphs, resulting in entity embeddings. At the same time, GCNCS takes the contextual information and the prior probability into account. These features collaborate to find the correct target. In a word, the key contributions in this paper are listed as follows:

• A novel strategy for building entity graphs that drastically explores the semantic space consistency among the candidate entities is presented. This not only reduces the time needed to build an entity graph, but also enhances the coherence of the finally built graph.
• The asymmetric graph convolutional network is used to learn entity embeddings, which improves discriminative signals of the entities by fully exploring the asymmetric structural features of the entity graphs. In addition, the final EL features combine the contextual information and the prior probability as well.
• Experiments with benchmark datasets demonstrate the superior performance of our approach compared with the state-of-the-art EL methods. Our experimental studies also illustrate the influences of the key features.

Here is the main structure of the paper. Section 2 introduces and compares recent EL approaches. The defining of the EL problem starts in Section 3. Section 4 describes the architecture of GCNCS and details the key modules. Section 5 evaluates the proposed model and some baselines, and also measures the roles of various configurations and features. Section 6 summarizes the results obtained through the different approaches and points out the conclusions and the future work.

2. Related Work

Entities are things in the objective world, which are unambiguous and the basic elements of the KB [14,15]. However, the mentions in a text may be ambiguous, which means a mention could denote several different entities in different contexts and an entity may be described by different mentions, which can prevent us from understanding the text properly. Since EL is mainly aiming to address the multiple meanings of the mention and diverse mentions with the same meaning, EL maps the mentions in a document into proper entities in a given KB. In the last several years many EL approaches have emerged, mainly including the collective entity linking method and the individual method. The first deals with all mentions simultaneously, considering the global interdependence between the mentions in the same document. The second covers mentions independently, ignoring the influence among the mentions.

Table 1. Comparative analysis of popular entity linking models.

Category	Model	Input	Knowledge Base	Mention		Entity
				na	ctx	tl	ds	enl	pr	cg
Individual entity linking	DBpedia Spotlight [16]	document	DBpedia Wikipedia	✓	✓	✓	✓	✗	✓	✗
	CNNContex [17]	document	Wikipedia	✓	✓	✓	✗	✗	✗	✓
	MemNet(C+L) [18]	document	Wikipedia	✓	✓	✓	✓	✗	✗	✗
	MPME [19]	document	Wikipedia	✓	✓	✓	✓	✗	✓	✗
Collective entity linking	AIDA [20]	document	Yago Wikipedia	✗	✓	✗	✓	✓	✓	✗
	Babelfy [21]	document	BabelNet	✗	✗	✗	✗	✓	✗	✗
	L2R-WNED [6]	document	Wikipedia	✗	✓	✗	✓	✓	✗	✗
	Deep-ed [22]	document	Yago Wikipedia	✗	✓	✓	✗	✗	✓	✗
	NCEL [23]	document	Wikipedia	✓	✓	✓	✓	✓	✓	✗
	SeqGAT [24]	document	Wikipedia	✓	✓	✓	✓	✓	✓	✗

summarizes the popular entity linking models. Due to limitations of space, “na” and “ctx” denote the mention’s surface form and the context, respectively. Correspondingly, “tl” refers to the entity title, “ds” refers to the entity description, “enl” refers to the semantic relatedness of the entities, “pr” refers to the prior popularity, and “cg” refers to the entity category. The symbol ✓ indicates that the corresponding model uses the feature, while ✗ means the feature is not used.

2.1. Individual Entity Linking

Early studies show that individual entity linking mainly depends on the similarity between the mention’s context and the entity description. Bunescu et al. [25] were the first to select the entity category and description from Wikipedia. Moreover, they ranked the entities based on the support vector machine, picking the one with the highest probability. Zheng et al. [26] tried to use learning to rank models in EL task, including Ranking Perceptron [27] and ListNet [28]. Their experiments showed that the learning to rank models brought about better results than classification models, and ListNet obtained the highest accuracy. However, those models mostly exploited hand-engineered features to compute context similarities, ignoring the semantic relations among words. DBpedia Spotlight [16] not only annotated text documents automatically, but also linked mentions to DBpedia based on various features such as the prior popularity, the context similarity, and the topic coherence.

Deep learning can learn multiple levels of distributed representations from the given documents and KBs, which pushes EL models. He et al. [5] utilized stacked denoising auto-encoders to learn the input document representation, but ignoring the entity description made the effect not obvious. Francis [29] used different kinds of topic information from multiple granularities. Vector representations were produced with CNNs, and then combined with sum pooling. CNNContex [17] leveraged the semantic representations of the mention, the context, and the entity, and embedded the positions of the context words to capture the distance between the context word and the mention. Memory networks [30] can convert the input into the internal features, and select the key features by jointly using reasoning components and long-term memory components. MemNet [18] could seek important information based on an attention mechanism and two external memories. Therefore, the proposed method had the ability to interact with the memory multiple times, and learn complex functions with multiple levels of composition. TypeCoAtt [4] was a type-aware co-attention model, which read entity type information to improve the co-attention mechanism. The model designed by Zhang et al. [31] for incomplete knowledge base contained three parts, which were encoder, co-attention, and decoder. Recently, mapping unstructured text and a structured knowledge base into the same semantic space has drawn increasing attention. Yamada et al. [32] jointly learned the embedding of words and entities by extending the skip-gram model. Particularly, the generated embeddings were combined with traditional EL features, and the GBRT was employed to learn to rank. Subsequently, MPME [19] considered that the same semantic space is affected by the ambiguity of mentions, and it was necessary to learn various sense embeddings for each mention. EAT [33] took into account anchor texts for representing words and entities in a unique space, which skipped the extra alignment strategy.

2.2. Collective Entity Linking

Graph structure denotes the complex relationship between things in the objective world, which provides new ideas for EL. Han et al. [34] proposed a collective inference algorithm based on the referent graph which contained the mentions and the entities. GLOW [35] treated the EL task as an optimization problem on the basis of local and global variants and analyzed the advantages and the disadvantages of them. AIDA [20] tackled EL as NP-hard optimization problems by enriching the weighted graph with more local features. Therefore, the best joint mention–entity mapping was approximated by computing a dense subgraph. Alhelbawy et al. [36] ranked nodes based on the PageRank algorithm and selected the final result with initial confidence. The accuracy of 85.79% on the AIDA dataset indicated the effectiveness of this algorithm. Guo et al. [6,37] considered that most approaches were not suitable for less popular entities. The constructed graph was based on semantic relatedness, which could overcome the feature sparsity issue. Babelfy [21] used BabelNet to create semantic signatures, and exploited random walks with restarts to weight the network’s edges. PPRSim [38] combined local and global features based on Personalized PageRank to filter out noise brought about by incorrect entities.

Specifically, Zwicklbauer et al. [39] not only integrated semantic entity and entity-context embeddings into a graph, but also introduced a topic node to represent all unambiguous and already-disambiguated mentions. Nguyen et al. [12] introduced a model which benefitted the EL task by jointly learning the local context similarities and the topic-related features. Globerson et al. [40] observed that a non-salient entity only depended on a small subset of mentions. The proposed multi-focal attention could capture some coherence information for local and pairwise scores. Ganea et al. [22] used differentiable inference on the combination of entity embeddings, an adaptive local score, and contextual attention. Yang et al. [41] could perform a better local search from past and future global information based on Beam Search with a gold path. Gradient tree boosting was first employed to optimize entity assignments for all the mentions. Because of noise, data sparsity, and incomplete knowledge base, Phan et al. [42,43] considered that not all mentions were densely related to each other. Pair-linking strategies were presented to iteratively select the pair with the highest confidence at each step of decision making. Le et al. [44,45,46] modeled multiple relations between mentions, and treated them as latent variables. What is more, unlabeled texts and the multi-instance matching method were used to create initial noisy labels. Then, the noise detection component could identify and exclude the wrong entities. Yamada et al. [47] utilized the new masked entity prediction to randomly mask entities when learning contextualized embeddings of words and entities based on BERT.

Recently, graph neural networks have made great progress [48,49]. The low dimensional vectors generated by graph representation learning contain rich semantic information of node and edge [50]. This also provides a promising research direction for future EL methods [51,52]. NCEL [23] took various features and entity graphs into the asymmetric graph convolutional network to explore structural information among entities, which improved the computation efficiency by performing on a subgraph of adjacent mentions. RLEL [53] considered the global linking as a sequence decision problem, and the reinforcement learning optimized the results based on the linked mentions. Following the models above, SeqGAT [24] introduced BERT to learn local features, which were the input of the graph attention network to capture the topic coherence of the mentions. GNED [54] applied graph convolutional network on the entity–word graph to generate enhanced entity embeddings, which were fed to a CRF for collective EL. Deng et al. [10] generated entity embeddings based on graph convolutional network for combining global semantic information and latent relation between the entities. In addition, multi-hop attention was used to improve the representation of the mention context. DGCN [55] utilized a dynamic graph convolutional network to capture various features, which benefited the quality of EL. CoGCN [56] leveraged contexts to enhance the entity representation. Xue et al. [57] introduced random-walk layers to model the semantic interdependence between the entities from external knowledge.

Some challenges still remain in the previous-introduced models. On the one hand, individual methods mainly depend on the similarity between the mention context and entity description, which ignores the relations between entities. On the other hand, most collective methods rely on the graph structure, which uses the PageRank algorithm to rank entities. However, this may lead to the problem of not making full use of neighboring node information and excessive noise in the graph. In addition, graph representation learning can leverage the structural information of nodes to optimize node vectors, and asymmetric graph convolutional network iteratively updates node representations by aggregating neighboring nodes and edges. This provides a new way to solve the problems above. Therefore, GCNCS is proposed based on the asymmetric graph convolutional network and the contextualized semantic relevance.

3. Preliminaries

3.1. Entity Linking

The EL task can be formalized by the five-tuple $T=<d,M,K,E,f>$, where $d\in D$ represents a document and K denotes background KB. d contains zero or more mentions. Assume the set of the identified mentions is $M=\left\{m_1,m_2,m_3,\cdots ,m_i\right\}$. $E=\left\{E_1,E_2,E_3,\cdots ,E_i\right\}$ denotes all the possible entities referred to by M. The goal is to find the correct entity for a mention based on relevant features. Otherwise, it will return NIL. This can be described by formula: $f:M\times K\to E\cup NIL$. For this study, the premise is that the mentions have been recognized in advance. For each mention $m_i\in M$, an entity set $E_i=\left\{e_{i1},e_{i2},e_{i3},\cdots ,e_{ij}\right\}$ can be generated according to the proposed method in [58], where j denotes the number of possible entities of $m_i$. To get a better linking strategy f, the key is to construct an entity graph. Assume $G=(v,\epsilon )$ is the corresponding weighted entity graph of d, where $v=\left\{\left(m_i,e_{i,j}\left)\right|\forall m_i\in M,\forall e_{i,j}\in E_i\right)\right\}$ stands for the set of the nodes and $\epsilon$ represents edges between the nodes. To enhance the semantic information in the graph, undirected edges are drawn between two entities of the different mentions. Let $relationship(v_p,v_q)$ denote the relatedness between $v_p=(m_i,e_{i,j})$ and $v_q=(m_t,e_{t,l})$. Therefore, $relationship(v_p,v_q)>0$ indicates $v_p$ and $v_q$ are relevant, and the weight of the edge is $relationship(v_p,v_q)$. The more accurate the edge weight is, the better the EL effect is.

3.2. Entity Graph

Graphs can intuitively describe the complex relationships between the entities, which are divided into four categories such as unweighted undirected graphs, unweighted directed graphs, weighted undirected graphs, and weighted directed graphs. Figure 2a indicates that the two nodes of an edge can be swapped, but not in Figure 2b. The weighted edges in Figure 2c,d capture the relatedness of two nodes. The bigger the weight is, the more closely related the nodes are. This section explores the graph construction from the different views.

3.2.1. Normalized Google Distance-Based Entity Graph

Normalized Google distance (NGD) measures relatedness between the entities on the basis of Google’s search results. Suppose the terms are $w_1$ and $w_2$. $f\left(w_1\right)$ and $f\left(w_2\right)$ represent the search results for the individual term, while $f(w_1,w_2)$ indicates the search results for the pair of terms. Then the relatedness between the terms can be measured using the formula:

$$\begin{array}{c}\hfill NGD(w_1,w_2)=\frac{max\left\{logf\left(w_1\right),logf\left(w_2\right)\right\}-logf(w_1,w_2)}{logT-min\left\{logf\left(w_1\right),logf\left(w_2\right)\right\}}\end{array}$$

(1)

where T denotes total search engine indexes. The combination of NGD and Wikipedia can make the relatedness between entities more accurate and complete. For $e_{i,j}$ and $e_{t,l}$, the relatedness is computed through the following formula.

$$\begin{array}{c}\hfill rls(e_{i,j},e_{t,l})=1-\frac{log\left(max\right(|E_j|,|E_l\left|\right))-log\left(\right|E_j\bigcap E_l\left|\right)}{log\left(\right|W\left|\right)-log\left(min(E_j,E_l)\right)}\end{array}$$

(2)

where $E_j$ and $E_l$ are the set of entities which can link to $e_{i,j}$ and $e_{t,l}$ correspondingly, and W is the set of all entities in Wikipedia. The edge weight $relationship(v_p,v_q)$ between $v_p$ and $v_q$ is replaced by $rls(e_{i,j},e_{t,l})$. Moreover, the higher value reflects the stronger relatedness, and the maximum value is 1. For the example given in Figure 1, the edge weights are computed through NGD, and the corresponding entity graph is illustrated in Figure 3.

3.2.2. Link-Based Entity Graph

The rich structure in the background KB provides additional information. Take Wikipedia as an example, its nodes correspond to the entities and its edges connect the pairs of the entities that are semantically related. Based on this, we can construct graphs within the following constraints: (1) hyperlinks exist between the Wikipedia articles of two entities; (2) there is an article with hyperlinks to both entities within a window of 500 words. When both of the conditions above are satisfied, it can be assumed that there is a strong semantic relatedness between the entities, and the edge weight is set to 1. Therefore, the link-based entity graph corresponding to Figure 1 is shown in Figure 4.

3.2.3. Embedding-Based Entity Graph

Both NGD and link-based methods depend on statistics, which leads to computational complexity. To reinforce semantic information of entities, it is essential to learn the entity embedding based on DistilBERT [59]. DistilBERT becomes a smaller, faster, and lighter model by reducing the number of layers, which can be finetuned with good performances. Compared to other variants of BERT, DistilBERT reduces the computation time. Different entity contexts can generate different entity embeddings. Therefore, entity embedding is closely related to context information, which can alleviate ambiguous problems. $v\left(e_{i,j}\right)$ and $v\left(e_{t,l}\right)$ denote the entity embeddings of $e_{i,j}$ and $e_{t,l}$, respectively. As shown in Formula (3), the semantic relatedness between the entities can be computed with cosine similarity. The edge weight between $v_p$ and $v_q$ is set to $sm(e_{i,j},e_{t,l})$. Therefore, the embedding-based entity graph corresponding to Figure 1 is shown in Figure 5.

$$\begin{array}{c}\hfill sm(e_{i,j},e_{t,l})=cos(v\left(e_{i,j}\right),v\left(e_{t,l}\right))\end{array}$$

(3)

4. Proposed Model

EL is challenging due to the name variation and entity ambiguity. However, traditional methods require tedious feature engineering. To extract the semantic relatedness between entities, the graph representation learning generates low dimensional vectors for nodes, which also benefits effective calculation and inference. In order to fully leverage the relations between the entities extracted from the background KB, this paper proposes a model called GCNCS, which combines an asymmetric graph convolutional network and contextualized semantic relevance. Specifically, this model not only exploits the entity–entity topic coherence in a document, but also learns entity embeddings via propagating the semantic information from neighboring entities. In addition, prior popularity and context similarity are still considered. Figure 6 shows our overall structure, which mainly contains three modules: contextualized semantic relevance with bidirectional encoder representation from transformers (BERT), entity embedding with asymmetric graph convolutional network, and entity selector with multiple features. More concretely, the contextualized semantic relevance leverages BERT to encode mention context and entity description, obtaining local semantic relevance. In the entity embedding module, the entity graph represents the possible dependencies between the entities. Therefore, the entity embedding mainly relies on neighboring nodes. To get the best EL result, the entity selector module combines contextualized semantic relevance, prior popularity, and entity embedding.

4.1. Contextualized Semantic Relevance with BERT

Since a mention’s surface form and entity title are generally short and the superficial characteristics vary relatively widely, EL cannot find sufficient evidence. However, context words may contain important information about a mention. For example, when “qualcomm” or “phone” appears in the surrounding context of mention “apple”, it indicates that the referent entity is “Apple Inc.” rather than “Apple”. Moreover, the background KB provides rich textual data for the entities, which helps the mentions to find the target entity via contextualized semantic relevance.

BERT overcomes long-range dependencies problem caused by LSTM and captures information from multiple perspectives, including lexical, syntactic, and semantic through a large-scale pre-training corpus. The deep bidirectional representations for mention and entity are achieved by fusing the left with the right context. The BERT model is first initialized with the pre-trained parameters, and all of the parameters are fine-tuned according to the needs of the EL task. BERT can learn both single sentence embeddings and sentence pairs embeddings. Here, the mention context and the entity description are concatenated as the input to BERT. Then, the output of the last layer $S_e$ is selected as the context similarity feature.

4.2. Entity Embedding with Asymmetric Graph Convolutional Network

The asymmetric graph convolutional network is applied to graph structure and can deal with data in a non-Euclidean space. Therefore, the topic coherence is achieved among the mentions within a document, and the node embeddings are iteratively updated. Note that the initialized state vectors of all nodes are node features or node attributes. Then, new node embeddings are generated by the combination of neighborhood aggregation and their own features. The main idea is the fusion of neighborhoods’ embeddings into node embeddings. Let $G=(v,\epsilon )$ be an undirected entity graph and let $v=\left\{v_1,\cdots ,v_n\right\}$ be a set of nodes, where n represents the number of nodes. Note that a $n\times n$ adjacent matrix A can represent the aforementioned entity graph, where the edge weight $a_{ij}$ is computed using a link-based method or embedding-based method. Moreover, the features of each node are initialized with entity embeddings learned by DistilBERT. The input set of node features is $\left\{h_1^0,\cdots ,h_n^0\right\}$, and the target output set is $\left\{h_1^L,\cdots ,h_n^L\right\}$ in the L-th layer. Suppose $h_i^l$ represents the embeddings of node i in the l-th layer, which is gained by aggregating the node embeddings from the (l-1)-th layer and its neighboring nodes. The formula is as follows:

$$\begin{array}{c}\hfill h_i^l=\sigma (\sum _{j=1}^n{a}_{ij}{W}^{\left(l\right)}{h}_j^{(l-1)}+b^{\left(l\right)})\end{array}$$

(4)

where $W^{\left(l\right)}$ and $b^{\left(l\right)}$ denote weights and bias in the l-th layer, and $\sigma$ is a non-linear activation. After numerious graph convolutions, the final node features can be acquired, which means the corresponding entity embeddings are represented by $G_e$.

4.3. Entity Selector with Multiple Features

The prior popularity $P_e$ is a statistical feature of entities, which represents the probability of mapping a mention to an entity. The entity with the highest prior popularity feature is considered to be linked correctly. The contextualized semantic relevance $S_e$ denotes the matching between the mentions and the entities at the semantic level, taking their background into account. The entity embedding $G_e$ exploits link structure and dependencies between the entities, which is also an intuitive mapping of topic coherence among all mentions within a document. Therefore, in order to improve the effectiveness of EL, it is necessary to consider a combination of the prior popularity, the contextualized semantic relevance, and the entity embedding. This paper suggests an entity selector for mapping the mentions correctly. For each mention m and its entity e, the final score function is defined in Equation (5).

$$\begin{array}{c}\hfill V_{m,e}=S_e\oplus P_e\oplus G_e\end{array}$$

(5)

where ⊕ denotes vector concatenation. The size of each vector dimension is the same. The concatenated vectors are then fed into a multi-layer perceptron, and which is also used as the input of a softmax to predict the similarity between the mention m and the entity e. The details of this process are given in Equations (6) and (7).

$$\begin{array}{c}\hfill f^l=Relu(W^l{f}^{l-1}+b^l)\end{array}$$

(6)

$$\begin{array}{c}\hfill sim(m,e)=softmax\left(f^l\right)\end{array}$$

(7)

where $f^l$ is the output of the last layer of MLP, and $f^{l-1}$ is the output of the $(l-1)$th hidden layer. $W^l$ and $b^l$ are trainable parameters and the bias. The entity with the highest similarity score is regarded as the target mapping entity.

5. Experiment

5.1. Experimental Settings

5.1.1. Datasets

For the purposes of evaluation of GCNCS, datasets with different properties are used, including AIDA [20], MSNBC [60], AQUAINT [61], ACE04 [35], and CWEB [6]. Our model is trained on AIDA, which contains 1393 news documents. AIDA is much larger than most EL datasets, and the average number of mentions per document is 19.9. The other four datasets are used to test the generalization ability of each model. MSNBC collects 20 documents from 10 different domains, and the average number of mentions per document is 32.8. AQUAINT annotates 50 documents, and the average number of mentions per document is 14.5. ACE04 contains 35 documents, and the average number of mentions per document is 7.1. The three datasets above have many popular entities, and their EL results mainly rely on prior probability. There are still challenges for less popular entities. Therefore, Guo et al. [6] created a new dataset, CWEB, by mixing documents with different levels of difficulty. CWEB is a subset of ClueWeb, and it has 320 documents. Its average number of mentions per document is 34.8. Figure 7 illustrates the prior popularity distribution over the entities in the test datasets.

5.1.2. Parameters

A Wikipedia dump on 7 April 2016 is used as the target knowledge base. The entity embeddings are initialized by using $DISTILBER{T}_{BASE}$, which has 6 layers and 6 heads. To learn the contextualized semantic relevance, the pre-trained uncased $BER{T}_{BASE}$ is utilized, which has 12 heads and 768 hidden states. In addition, the entity graph is constructed based on embeddings. The number of asymmetric graph convolutional network layers is set to 3, the learning rate is 1e-3, and the MLP consists of 3 layers.

5.1.3. Complexity Analysis

The module of contextualized semantic relevance gets the context similarity by finetuning BERT. Therefore, the complexity of GCNCS is mainly determined by the entity embedding module. Traditional graph-based models have a high time complexity. A well-known approach, proposed by Hoffart et al. [20], has a complexity of $O\left(k^3{n}^3\right)$, where k denotes the average number of mentions per document. GCNCS has a lower complexity of $O(T\ast k^2{n}^2)$, where T is the number of asymmetric graph convolutional network layers.

5.2. Evaluation Metric

Following the previous research [4,62], GCNCS only considers non-NIL mentions which correspond to correct entities in KB. Therefore, micro-averaged accuracy is selected for evaluation, which only concerns the correctly linked mentions. The definition of the evaluation metric is given in Equation (8).

$$\begin{array}{c}\hfill MicA=\frac{\left|correctlylinkedmentions\right|}{\left|totalnumberofallmentions\right|}\end{array}$$

(8)

5.3. Result and Discussion

This section details the results of GCNCS by comparing them with the state-of-the-art graph-based models and the prior popularity method. These models are briefly introduced as follows:

Prioris derived from Wikipedia entity hyperlinks, and can easily find the most likely entity to a mention, which relies on a direct ranking of the entities.

RI [63] leverages external knowledge from Wikipedia to extract relations between the mentions. Note that this method builds relation constraints and performs reasoning.

AIDA [20] integrates popularity priors, similarity measures, and coherence into a framework. Dense subgraphs are constructed to approximate the most promising linked entities. However, this greedy algorithm is time-consuming.

DoSeR [39] exploits the semantic embeddings by constructing a disambiguation graph twice. In addition, this method does not depend on any KB, and is more suitable for large datasets.

Table 2. The MicA comparison of various baselines with GCNCS.

Model	MSNBC	AQUAINT	ACE04	CWEB
Prior	0.89	0.83	0.84	0.70
AIDA [20]	0.79	0.56	0.80	0.58
RI [63]	0.90	0.90	0.86	0.68
DoSeR [39]	0.91	0.84	0.91	-
GCNCS	0.91	0.90	0.94	0.74

presents the overall performance of various models on each dataset. Compared with Prior, GCNCS improves the MicA by 0.02 on MSNBC, 0.07 on AQUAINT, 0.1 on ACE04, and 0.04 on CWEB. GCNCS significantly outperforms all graph-based models on ACE04 and CWEB due to the short documents and the lower prior popularity feature. The results reveal the superiority of GCNCS for unpopular entities. It also means that our model not only captures deep semantic features between the mentions and the entities, but also generates the entity embeddings containing more neighbors and asymmetric graph structure information in the entity graph.

5.4. Impact of Different Modules

To explore the effects of contextualized semantic relevance and entity graph, we analyzed the variants of GCNCS in two aspects.

(1) Influence of contextualized semantic relevance

GCNCS mainly utilizes BERT to learn deep semantic features from the mention’s context and the entity description. However, The bi-directional long short-term memory network (BiLSTM) is widely used in common NLP tasks, which can capture both previous and future contextual semantic information. To evaluate the performance of different computations for the contextualized semantic relevance, BERT is compared with BiLSTM in two ways. On the one hand, neglecting the entity embedding feature and the prior popularity feature, we just use BiLSTM and BERT to learn the contextualized semantic relevance. On the other hand, GCNBL replaces BERT in GCNCS for contextualized semantic relevance with BiLSTM. Comparative results in

Table 3. The MicA comparison of variants of GCNCS with different computations for the contextualized semantic relevance.

Model	MSNBC	AQUAINT	ACE04	CWEB
BiLSTM	0.56	0.44	0.62	0.48
BERT	0.74	0.57	0.77	0.58
GCNBL	0.78	0.77	0.82	0.71
GCNCS	0.91	0.90	0.94	0.74

illustrate that the models based on BERT obtain better performance on all the datasets. All in all, MicA has increased by 0.14 on average. In particular, GCNCS has achieved about 0.1 MicA improvement on average, indicating that BERT can obtain more valuable information from the mention’s context and entity description. Apart from it, the results also show that even with a powerful language model, MicA is only around 0.6 if we only leverage the local semantic feature. Therefore, it is necessary to combine global features, such as the prior popularity feature and entity embedding feature.

(2) Influence of entity graph

The entity graph is the basis for graph-based EL models. Various graph construction methods reveal the strength of relatedness between entities from different perspectives. The more accurate the weights of the edges between the entities are, the better the linking performance is. Since the Normalized Google distance-based entity graph utilizes the statistical features to compute edge weights, it does not contain semantic information. Therefore, we observe the impacts of the link-based entity graph and embedding-based entity graph on the linking result. In

Table 4. The MicA comparison of variants of GCNCS with different constructions for the entity graph.

Model	MSNBC	AQUAINT	ACE04	CWEB
GCNLJ	0.69	0.56	0.72	0.52
GCNEB	0.83	0.78	0.84	0.65
GCNLR	0.84	0.84	0.90	0.67
GCNCS	0.91	0.90	0.94	0.74

, GCNLJ applies asymmetric graph convolutional network to link-based graph for the linking inference. GCNEB uses the asymmetric graph convolutional network on an embedding-based graph to find the correct entity. GCNLR and GCNCS are based on GCNLJ and GCNEB, respectively, combining the prior popularity and entity embedding. The results demonstrate that the embedding-based graph can capture more semantic information between the entities. The overall effects are superior to those of the link-based graph.

6. Conclusions

EL is beneficial for people to understand any text, due to the ambiguity of mentions. Existing models often ignore the coherence among entities and focus more on popular entities. To overcome the limitations mentioned above, this paper proposes GCNCS, which collaboratively combines the entity embeddings, prior popularity, and context similarity. Specifically, an entity graph is constructed to denote the dependencies among entities, indicating the topic coherence of mentions. Then, the asymmetric graph convolutional network produces entity embeddings by propagating the semantic information from neighboring entities. Moreover, the contextualized semantic relevance module has a symmetrical structure, which encodes mention context and entity description to achieve local semantic relevance based on BERT. Compared with the state of the art, GCNCS successfully finds the correct entity by employing local and global features. Experimental results demonstrate that GCNCS has superior performance in terms of MicA on four datasets.

Our study not only provides a new research perspective on EL, but is also the foundation of natural language understanding, which can create hyperlinks to Wikipedia for any kind of input text. Take search queries as an example. EL extends entity awareness in Web and news searches by understanding and expanding phrases that refer to semantic types.

Although some key issues have been addressed in this paper, there are still some limitations. GCNCS is on the basis of Wikipedia. It works well for news articles and other general-domain texts on the Web, but not for domain-specific texts such as research papers or novels. This is because Wikipedia can provide textual descriptions and relational information for the general-domain entities. The domain-specific entities only have a short descriptive sentence. In the future, we plan to explore EL tasks on multimodal data. The image information is helpful for finding correct entities. Entity recognition is the basis for EL. Combining the two tasks will enforce the effect of them.