Enhanced Chinese Domain Named Entity Recognition: An Approach with Lexicon Boundary and Frequency Weight Features

Abstract

Named entity recognition (NER) plays a crucial role in information extraction but faces challenges in the Chinese context. Especially in Chinese paleontology popular science, NER encounters difficulties, such as low recognition performance for long and nested entities, as well as the complexity of handling mixed Chinese–English texts. This study aims to enhance the performance of NER in this domain. We propose an approach based on the multi-head self-attention mechanism for integrating Chinese lexicon-level features; by integrating Chinese lexicon boundary and domain term frequency weight features, this method enhances the model’s perception of entity boundaries, relative positions, and types. To address training prediction inconsistency, we introduce a novel data augmentation method, generating enhanced data based on the difference set between all and sample entity types. Experiments on four Chinese datasets, namely Resume, Youku, SubDuIE, and our PPOST, show that our approach outperforms baselines, achieving F1-score improvements of 0.03%, 0.16%, 1.27%, and 2.28%, respectively. This research confirms the effectiveness of integrating Chinese lexicon boundary and domain term frequency weight features in NER. Our work provides valuable insights for improving the applicability and performance of NER in other Chinese domain scenarios.

1. Introduction

Over the past few years, with the development of the Internet and information technology, various domains have accumulated a large amount of textual data, containing valuable information and knowledge [1]. Effectively utilizing this textual data can have a significant impact on various domains. Information extraction refers to automatically extracting key information from massive data, named entity recognition (NER), as a subtask, and has become a research hotspot in various domains [2].

In the domain of education, popular science (also known as popsci) is a scientific explanation targeted at the general audience [3]. Popular science education can cultivate the public’s interest in science and promote the popularization and dissemination of science and technology in society, thereby fostering social development. Faced with the growing data in the domain of popular science education, how to quickly and accurately identify keywords and reveal deep semantic relationships is an urgent issue for intelligent systems in popular science education. Many discoveries and research results in the domain of paleontology contribute to understanding the evolution of the Earth, as well as the origin and evolution of life and humanity [4], and one of the advantages of paleontology is popularization. In order to promote the latest research results to the public and advance the development of popular science education, the efficient extraction of key information from texts in the domain of paleontology popular science using deep learning-based information extraction technology has gained widespread attention from domain experts. One of the primary tasks in the domain of Chinese paleontology popular science NER is to identify different types of entities, such as a person, geological time, strata, and fossil, etc., from unstructured textual data. NER involves preprocessing, feature extraction, and classification to identify entities from unstructured textual data. By recognizing entities, valuable information can be extracted from textual data, laying the foundation for subsequent tasks [5], such as relationship extraction, sentiment analysis [6,7], and knowledge graph construction.

The NER methods can be summarized from rule-based methods to deep learning-based methods. In the early stage, rule-based methods utilized predefined and inductive assumptions and rules to identify entity names in the text and classify them [8,9]. However, with the increase in annotated data, rule-based methods need constant updates to discover more entities. Traditional machine learning methods, mainly based on supervised learning, involve training classification or sequence labeling models on annotated datasets to learn features of positive or negative instances, or type features. For example, based on a support vector machine (SVM) [10], hidden Markov model (HMM) [11,12], and conditional random field (CRF) [13], etc., those methods no longer require the manual construction of rules, but they still require feature selection. In recent years, with the development of deep learning, various neural networks have been used to address NER problems, such as convolutional neural networks (CNNs) [14], recurrent neural networks (RNNs), such as long short-term memory (LSTM) [15] and bidirectional LSTM (BiLSTM) [16], graph neural networks (GNNs) [17], and attention mechanisms [18], etc. Neural network methods typically use vectors to learn the syntax and context information of language. Pretrained models, represented by BERT (bidirectional encoder representation from transformers) [19], have a significant advantage in extracting contextual information features from text due to training on large-scale datasets. This notably improves various metrics for natural language processing (NLP) tasks. In the field of NER, significant achievements have been made by methods based on fine-tuning pretrained models, like BERT [19,20], BERT-CRF [21,22], and BERT-BiLSTM-CRF [23]. Pretrained models can learn contextual features of text on the basis of training on large-sample data, enabling fine-tuning in situations with limited annotated data and subsequently learning contextual features for downstream tasks.

The models mentioned above are based on sequence labeling to accomplish NER, which means performing a multi-classification task for each token. However, in the domain of Chinese paleontology popular science, as shown in Figure 1, the entity “古脊椎动物与古人类研究所” (Institute of Vertebrate Paleontology and Paleoanthropology, Chinese Academy of Sciences) consists of 12 Chinese characters; the span “寒武纪地层” (Cambrian Strata) includes the Geological Time entity, the Cambrian, and the Strata entity, Cambrian Strata; the span “古生物学家Grant Zazula” (the Paleontologist named Grant Zazula) contains both Chinese and English characters. Due to the existence of the mentioned phenomena and problems, directly applying sequence labeling models will significantly degrade the model’s performance, and these problems directly hinder the further development of NER in this domain. To address this issue, the pointer labeling task is proposed. Pointer labeling is a framework for NER. It sets start and end pointers to record the starting and ending positions of each entity, and simultaneously annotates the entity type. This method excels in recognizing long entities and nested entities. However, using only the pointer labeling for NER results in the loss of Chinese lexical boundary information, causing the model to erroneously output any two boundaries as entities. Recently, the integration of lexicon-level features into Chinese NER has attracted widespread attention. Integrating lexicon-level features involves incorporating lexicons as external information, helping to determine the span and type of entities. At the same time, in our domain, we found that the distribution of domain-specific lexicon also affects the performance of NER. Currently, the utilization of distributional features of a domain-specific lexicon is still in its early stages.

To address the above-mentioned issues, in this paper, we propose an enhanced NER method based on lexicon boundary and frequency weight features, named the BERT-Lexicon-PointerNetwork (BERT-LPN) model. The model is proposed for extracting Chinese paleontology popular science entities and it consists of an encoder based on BERT, a lexicon feature fusion layer and a fully connected classifier layer from the bottom up: the encoder is a Chinese character-level model based on BERT, and it maps Chinese characters to a low-dimensional, highly dense real-vector space to extract latent semantic information from Chinese entity elements; the lexicon feature fusion layer takes lexicon boundary and weight information from Chinese word segmentation as input to capture the lexicon-level features of the input; finally the fully connected classifier layer takes BERT-encoded embeddings along with lexicon-level features as input to generate start position and end position labels for entities. The experimental results indicate that our proposed model outperforms previous models, and our approach achieves state-of-the-art (SOTA) performance on the constructed dataset.

The main contributions of this research are summarized as follows:

• Based on the characteristics of texts in the domain of Chinese paleontology popular science, we propose a NER model enhanced with Chinese lexicon boundary and frequency weight features. This model is designed for identifying and extracting entities from unstructured textual data.
• Based on the structural characteristics of our model, we propose a data augmentation method utilizing all entity types and sample entity types to alleviate the inconsistency between training and prediction tasks.

To comprehensively evaluate the performance of our model, we established a new dataset, namely the PPOST dataset, specifically designed for NER in the Chinese paleontology popular science domain. The dataset primarily consists of data from authoritative institutions in the domain of paleontology popular science in China. Experimental results on both public datasets and the PPOST dataset validate the effectiveness of our approach. In addition, we also analyze the possibility of applying our method to other domains, providing a reference for enhancing NER performance in other domains.

The remaining sections of this paper are organized as follows. Section 2 introduces NER work based on deep learning that is closely related to our work. Section 3 presents our proposed model and core algorithms. Section 4 provides a detailed description of the datasets used in our experiments and analyzes the experimental results. Finally, we conclude the article.

2. Related Work

In this section, we introduce works closely related to our proposed model, including NER models that integrate lexicon-level features, methods for addressing domain data scarcity, and NER models based on pointer labeling, as follows.

2.1. Lexicon-Based Chinese NER

Chinese, unlike languages, such as English, that use spaces as separators, encounters the issue of word segmentation, where Chinese takes words as the basic semantic units. To better adapt to Chinese NER, recent studies have found that introducing Chinese lexicon-level features can enhance the performance of NER models. For example, Zhang et al. [24] proposed a lattice LSTM model for Chinese NER. This model is an extension of character-based NER models, incorporating words as input and additional gates to control information flow. The model explicitly utilizes word and word order information. However, due to the RNN structure adopted by the lattice LSTM model, it cannot capture long-distance dependencies, and the introduction of lexical information is lossy. To address these limitations, Li et al. [25] proposed the FLAT model for Chinese NER. This model is based on a transformer [26] to solve the lexical loss problem in lattice LSTM and uses relative position encoding to adapt a transformer to NER, enabling efficient GPU parallel computation. These methods, combined with pre-trained language models, explore Chinese NER and achieve SOTA performance on several Chinese benchmark datasets. However, Guo et al. [27] and Liu et al. [28] believe that existing lexicon-based models only conform to lexicon features through shallow and randomly initialized encoding layers without integrating them into the underlying layers of pre-trained language models to explore deep lexicon knowledge. To address this, they proposed methods to deeply integrate external lexicon features into the pre-trained language model BERT, achieving more effective fusion of entity boundaries and lexical information.

All of these studies have explored the integration of Chinese lexicon-level features, but they did not address the practical issues faced by more detailed domains. Moreover, these methods rely on the quality of lexicons and have certain requirements for computational resources. In contrast, this paper focuses on extracting entities from the Chinese paleontology popular science domain, integrating Chinese lexicon-level features in a simple and effective manner while enhancing relative positional information. It provides a reference for addressing the practical issues faced by this domain.

2.2. Prompt and Pointer Network

Typically, there is not much annotated data for downstream tasks, and there is often a gap between pre-trained BERT models on large corpora and fine-tuning on downstream tasks. To address the gap, some researchers have proposed modeling downstream tasks in the form of pre-training tasks [29]. Taking BERT as an example, some tasks can be modeled as masked language modeling (MLM) or next sentence prediction (NSP), and this approach is particularly effective in scenarios with small datasets [30]. For instance, Cui et al. [31] and others proposed the TemplateNER method, which completes NER using prompt learning. This method employs a template-based approach to solve the problem of few-shot NER: a template is pre-defined, for example, “<candidate_span>is a <entity_type> entity”. With the help of a suitable prompt, this method reduces the difference between pre-training and fine-tuning, allowing the model to achieve good results with a small number of samples. It significantly outperforms traditional sequence labeling methods and distance-based few-shot NER methods in cross-domain and few-shot scenarios. Subsequently, methods based on the prompt mechanism have been widely used in NER [32,33], such as modeling the task as machine reading comprehension and question answering (MRC-QA) format, fitting the NSP task for NER [34].

The NER model based on the dual pointer network aims to address both the challenges of recognizing long entities and nested entities. Lu et al. [35] proposed a unified extraction framework, where the NER module is based on a pre-trained model, employing a multi-layer dual pointer network. Input samples are constructed based on the template “[SPOT]<entity_type>[text]”, predicting the start and end positions of possible entities in the samples, and NER is completed through the decoding algorithm. However, real-world datasets often exhibit a long-tail distribution problem, with a few types representing the majority of the samples. The multi-layer dual pointer network requires setting up a classifier for each entity type, potentially leading to some classifiers not being adequately trained, thus impacting the overall performance of NER model.

To mitigate this issue, some researchers proposed a model based on a single-layer dual pointer network and the prompt mechanism. Gong et al. [36] introduced a model based on a single-layer dual pointer network, constructing the input in the format “<entity_type>[SEP]<text>” to accomplish NER. However, Su et al. [37] argue that the conventional design of the dual-pointer network, when performing NER or MRC, typically employs two modules to separately identify the start and end positions of entities. This can lead to inconsistencies during training and prediction. To address this issue, they proposed global pointer model which uses a globally normalized approach for NER. In non-nested scenarios, it achieves results comparable to CRF, and in nested scenarios, it performs well. Moreover, it allows for fully parallelized training, significantly accelerating the training process. This method addresses the inconsistency between training and prediction tasks from a model perspective.

All these studies have explored NER methods based on pre-trained models and proposed unique insights and solutions for various issues. However, most of them optimized from the perspective of the model without considering the scarcity and differences in datasets across various domains in real-world applications. In contrast, this paper introduces a data augmentation method that enhances NER performance by increasing training data, alleviating the inconsistency between training and prediction tasks. This approach provides a reference for improving NER performance in domain-specific applications.

3. Methods

In this section, we provide a detailed description of the proposed model architecture, and the overall structure of our model is illustrated in Figure 2. Firstly, we introduce some preprocessing operations specific to this domain to acquire the necessary experimental foundation. Secondly, based on the WordPiece [38] algorithm, we use BERT to obtain character-level embeddings and acquire lexicon-level features through Chinese word segmentation; lexicon-level features fusion are accomplished using a multi-head self-attention mechanism. Finally, the embeddings based on BERT and the lexicon-level features serve as inputs to the dual pointer network, which is applied to complete the NER through decoding in the domain of Chinese paleontology popular science.

3.1. Preprocessing

To obtain domain term frequency weight features, we base our calculations on the collected corpus in the domain of Chinese paleontology popular science. TF-IDF [39] is a statistical method that consists of term frequency (TF) and inverse document frequency (IDF) and evaluates the importance of a term to a document set or one of the documents in a corpus, the TF-IDF scores of domain terms are computed as domain term frequency weights in our approach. Specifically, for the raw text corpus we collected, we employ the Chinese word segmentation tool pkuseg [40] for segmentation. After removing stop words and segmenting the Chinese text, the TF-IDF scores of each term are calculated as the weight feature for domain term frequency. Since there are $N$ documents in total, term $t$ has $N$ TF-IDF scores, and we integrate these scores using the averaging strategy because this approach preserves more features, thereby better assessing the weight of a term in the corpus and obtaining the final feature for domain term frequency. The method for calculating domain term frequency weights is as follows:

$$W_{t,d}=t{f}_{t,d}\cdot log(\frac{N}{1+d{f}_t}),$$

(1)

$$Weigh{t}_t=\frac{{\sum }_{i=0}^N{W}_{t,d}^i}{N},$$

(2)

where $t{f}_{t,d}$ represents the term frequency of term $t$ in document $d$, $log$ is the natural logarithm, $d{f}_t$ represents the number of documents out of $N$ that contain term $t$, $W_{t,d}$ is the TF-IDF score of term $t$ in document $d$, and $Weigh{t}_t$ represents the average weighted domain term frequency weight of term $t$. The final domain term frequency weights are shown in

Table 1. Domain term frequency weight lookup table (DTFW-lookup Table).

Domain Term	Weight
恐龙 (Dinosaur)	W恐龙 = 1.36 × 10−3
侏罗纪 (Jurassic)	W侏罗纪 = 3.67 × 10−4
中国科学院 (Chinese Academy of Sciences)	W中国科学院 = 7.38 × 10−5
…	…
Termi	Wi = x

.

Usually, the NER based on a dual pointer network and the MRC-QA mechanism faces the problem of inconsistency between training and prediction tasks. That is, during training, fitting is carried out with the correct entity type as a prompt, while in prediction tasks, it is often not possible to specify the correct prompt for prediction, meaning the prediction task also includes an entity classification task. To alleviate this inconsistency and enhance the model’s generalization ability, we propose a data augmentation method based on the set difference between all entity types and sample entity types.

As shown in Figure 3, our dataset includes 13 types of entities, denoted as the set $AllTypes$ = {Strata, FossilClassification (FossilClass), Subject, Event, Position, Time, Feature, Person, Organization, GeologicalTime (GeoTime), Address, Fossil, BiologicalClassification (BioClass)}, while the current set of entity types included in the samples is denoted as set ${Gold}_{sample}$. Through set difference operations on the sets, the entity types not included in the current samples are generated. In this way, after calculating the difference set for all samples, the labels of the enhanced data are set to an empty set to avoid introducing noisy data, as follows:

$${Diff}_{sample}=\{type\in AllTypes:type\notin {Gold}_{sample}\},$$

(3)

$$D{A}_{sample}^{item}=\{Tex{t}_{sample},EntityType={Diff}_{sample}^{item}, Label=Ø\},$$

(4)

where ${Diff}_{sample}^{item}$ is an element in the collection ${Diff}_{sample}$, and $D{A}_{sample}^{item}$ is a piece of data generated when the entity type is ${Diff}_{sample}^{item}$.

Then, based on the given predefined data augmentation ratio, a certain number of augmented data are generated through a random sampling algorithm and merged into the training dataset for model training. By introducing augmented data, the model’s perception of entity types not present in the sample data is strengthened, effectively alleviating the inconsistency between the training and prediction tasks.

3.2. BERT Encoder

To effectively capture textual information from text, the encoder is constructed based on a pretrained language model, BERT. Given an input (i.e., a sentence or a word) $S=[s_1,s_2,\dots ,s_{\left|S\right|}]$, the encoder tokenizes it using a predefined vocabulary and encodes $s_i$ into a vector representation $\overline{s_i}$. For detailed steps, please refer to the literature [19]. The encoder embeds the text $S$ into vector representations ${\mathsf{\Gamma }}^{\mathrm{S}}={\mathbb{R}}^{len\times dim}$, where $len$ represents the maximum acceptable input length for the model, $dim$ represents the dimension of BERT token embedding vectors, and in the case of BERT-base dim is set to 768. Generally, ${\mathsf{\Gamma }}_{\mathrm{j}}^{\mathrm{S}}$ can be considered as the embedding of the ${\mathrm{j}}^{\mathrm{t}\mathrm{h}}$ token, and it is worth noting that due to the usage of the WordPiece tokenization algorithm, variations may occur in our methods. Unlike other Chinese NER models, we use the WordPiece tokenization algorithm to represent Chinese characters as independent tokens and English as their subword representations, and the method can simplify the model input and reduce the complexity of model. Additionally, based on BERT, we fine-tune the model in conjunction with the MRC-QA task. The detailed steps are as follows:

$$input=“<entity\_type>,<text>",$$

(5)

$$tokenized=WordPiece\left(input\right)=\{t_1,t_1,\dots ,t_n\},$$

(6)

$${\chi }_i=e\left(t_i\right).$$

(7)

In the above, using the provided $text$ sequence and its corresponding entity type $entity\_type$, we construct the $input$ and obtain $tokenized$ representations for the input using the WordPiece algorithm. The variable ${\chi }_i$ represents the embedding vector by BERT for the given $tokenized$ representation.

3.3. Lexicon Encoder

To better harness Chinese lexicon-level information, we have introduced lexicon boundary and frequency weight features. Given the input, it is segmented into individual lexicons or characters after applying a segmentation tool, as follows:

$$S{egId}_{input}=\left\{\right(0,a),(a+1,b),\dots ,(x,y),\dots ,0\dots ,0\},$$

(8)

where $(x,y)$ represents the segment composed of the characters from the x^th to the y^th position in the input sequence, in other words, in the ordered set of $S{egId}_{input}$, each element represents a Chinese lexicon or character. Let us define a function $f(i,j)$—element i repeats j times results in an ordered set, as follow:

$$f(i,j)=\{i repeats j times\},$$

(9)

giving $(x,y{)}_{id}\in SegI{d}_{input}$, where $id$ is the index of $(x,y{)}_{id}$ in the ordered set $SegI{d}_{input}$, and we substitute$f(i,j)$ to obtain$f(id,y-x)$, where $y-x$ is the length of the current lexicon or character segment. In summary, we obtain the lexicon boundary features of the input sequence through Algorithm 1, and its time complexity is O$\left(n\right)$, where $n$ is the length of the input sequence. At the same time, if $n$ does not reach the maximum allowed length of the model, padding with zeros is applied after the last element of $boundary$ until it reaches its maximum. On the other hand, as shown in Figure 4, when there are three entities which belong to the same type (BioClass) in a given input (lexicons 0, 2, and 4), without considering relative positional information, the model may arbitrarily combine their boundaries in the output. For example, the phrase “恐龙位于爬行动物和鸟类” (Dinosaurs are between reptiles and birds, with characters between the start of lexicon 0 and the end of lexicon 4) could be incorrectly combined into a single BioClass entity. With the inclusion of boundary information, the model becomes sensitive to the length and span of entities.

Algorithm 1 Lexicon Boundary Generation

Input: $Ordered Set of {SegId}_{input}$

Output: $Ordered Set of Lexicon Boundary$

1: $boundary=\left\{\right\}$

2: for each ${(x,y)}_{id}\in {SegId}_{input}$ do

3:        $item=\left\{\right\}$

4:        for each $index\in [0,y-x]$ do

5:               $item=item\cup \left\{i\right\}$

6:        end for

7:        $boundary=boundary\cup item$

8: end for

9: return $boundary$

Given the ${boundary}_i$, we look up the boundary index embedding $e_i$ from the specialized embedding matrix. This matrix contains fixed-size embeddings for each boundary index 0, 1, 2, …, and these embeddings are learned through backpropagation.

Similarly, $(x,y{)}_{weight}\in SegI{d}_{input}$, where $weight$ represents the term frequency weight of the x^th to y^th subsequence of the input, and this weight value is obtained through the DTFW-lookup Table in Section 3.1. Substitute$f(i,j)$ to obtain$f(weight,y-x)$, where$y-x$ is the length of the current segment. This step will result in the lexicon or character frequency weight representation $w_i$ for the input. Specifically, when the lexicon or character do not exist in the DTFW-lookup Table, we use zero as the default value, as follows:

$$w_i=\{\begin{array}{ccc}{DTFW-Lookup Table}_{term},& & if term \in DTFW-Lookup Table\\ 0,& & else.\end{array}$$

(10)

Ultimately, for the given ${boundary}_i$, the lexicon-level feature representation at position$i$ is as follows (whereas $\circ$ denotes concatenation):

$${\theta }_i=e_i\circ w_i.$$

(11)

Simultaneously, in order to capture features of different dimensions for boundaries and weights, we employ a multi-head self-attention mechanism for feature fusion. The workflow of the above algorithm is illustrated in Figure 5.

In this mechanism, the multi-head self-attention mechanism calculates attention weights and outputs through the query (Q), key (K), and value (V) inputs. Specifically, for the inputs Q, K, and V, the calculation formula for the output vectors is as follows:

$$MultiHead(Q,K,V)=hea{d}_1\circ hea{d}_2\circ \dots \circ hea{d}_h\times W^o,$$

(12)

where $W^O\in {\mathbb{R}}^{h{d}_v\times d_{model}}$, and $hea{d}_i$ is calculated as follows:

$$hea{d}_i=Attention(Q{W}_i^Q,K{W}_i^K,V{W}_i^V),$$

(13)

$$where Attention(Q,K,V)=softmax(\frac{Q{K}^T}{\sqrt{d_k}})V,$$

(14)

where $W_i^Q\in {\mathbb{R}}^{d_{model}\times d_k}, W_i^K\in {\mathbb{R}}^{d_{model}\times d_k}, W_i^V\in {\mathbb{R}}^{d_{model}\times d_v}$($h$ represents the number of heads, and $d_{model}$ is the dimension of the input vector. Generally, $d_k$ = $d_v$ = $d_{model}$/$\mathrm{h}$).

3.4. Classifier

To address the challenges posed by long entities and nested entities, we apply a dual pointer fully connected network classifier. The classifier consists of two pointers: one indicating the start position of an entity and the other indicating the end position. In addition, a sigmoid function is applied to map the fully connected network output into the [0, 1] range. Finally, based on a specified $threshold$, the output is discretized into {0, 1} for representation, where “1” indicates that the position is the start position or end position of the entity, and “0” indicates the opposite. The specific classification process is illustrated in Figure 6. Given the embedding vector representation of the input, $context$, is determined as follows:

$$contex{t}_i={\chi }_i\circ {\theta }_i.$$

(15)

We utilize the $context$ vector as the input for the classifier and decode based on predefined $threshold$, as follows:

$$output=classifier\left(context\right),$$

(16)

$${position}_i^s,{position}_i^e=\{\begin{array}{ccc}1,& & if positio{n}_i>threshold\\ 0,& & else,\end{array}$$

(17)

where ${position}_i^s$ denotes the start position classifier’s output at position i, and ${position}_i^e$ denotes the end position classifier’s output at position i.

3.5. Loss Function

The loss function is applied to gauge the performance of the model, specifically the disparity between the model’s predictions and the actual targets. This discrepancy is typically referred to as the loss value, and the objective of the loss function is to minimize this value. When the model’s predictions perfectly align with the targets, the loss value is zero, indicating optimal model performance. We apply the binary cross-entropy loss (BCELoss) function to compute the disparity between the model’s outputs and the actual labels.

Additionally, the AdamW [41] optimizer, which is a variant of the Adam [42] optimizer, introduces a weight decay term to control the regularization of model parameters. Weight decay helps prevent overfitting by encouraging the model to use simpler parameter settings, thereby enhancing generalization capabilities. It is important to note that we do not explicitly introduce a regularization term in the loss function. In our loss function, we calculate the difference between the model’s outputs and the true labels using binary cross-entropy, and we apply the AdamW optimizer with weight decay to enhance the model’s generalization without explicitly introducing a regularization term in the loss function, as follows:

$$\mathcal{L}oss=\frac{1}{2}({\mathcal{L}}^s+{\mathcal{L}}^e),$$

(18)

where ${\mathcal{L}}^s$ denotes the start position classifier’s loss and ${\mathcal{L}}^e$ denotes the end position classifier’s loss. A sentence consists of $N$ tokens, $\mathcal{L}$ is calculated as follows:

$$\mathcal{L}(Y,P)=-\frac{1}{N}\sum _{i=1}^N[{\mathcal{Z}(y}_i,p_i)+\mathcal{Z}(1-y_i,1-p_i)],$$

(19)

$$\mathrm{where} \mathcal{Z}(y,p)=y\ast log\left(p\right),$$

(20)

where $log$ is the natural logarithm.

4. Experiments

4.1. Preparation

4.1.1. Dataset

We evaluated our proposed method on the following datasets: Resume [24], Youku [43], DuIE [44], and our own PPOST dataset.

The Resume dataset is a Chinese NER dataset focused on resumes. Youku provides an open NER dataset in the entertainment domain. DuIE is a large-scale manually annotated dataset, and all sentences in this dataset are extracted from Baidu Baike and Baidu News Search. The text in this dataset covers various domains found in real-world applications, including news, entertainment, and user-generated content. Due to the large size of the dataset, we randomly sampled a portion of the data, referred to as SubDuIE, for evaluation.

Table 2. Statistics on the four dataset for the experiments.

Dataset	Resume	Youku	SubDuIE	PPOST 1
Train	3821	8001	19,521	13,124
Eval	463	1000	1950	1396
Test	477	1001	1948	1395
Types	8	9	13	13
Avg 2	33	18	66	83

¹ The PPOST dataset includes 9% of data generated using the data augmentation method; ² “Avg” represents the average number of characters in the samples of this dataset.

shows the details of these datasets.

The PPOST dataset consists of manually labeled entity data in the domain of Chinese paleontology popular science. The label distribution of the PPOST dataset is illustrated in the Figure 7. It can be seen that even after data augmentation processing, the phenomenon of a long-tail distribution still exists in practical production scenarios.

4.1.2. Evaluation Metrics

To evaluate the performance on Chinese NER, following most of the baselines, we use precision (P), recall (R) and F1-score (F1) as the metrics, they are computed on the entity-level number of true positives (TP), false positives (FP) and false negatives (FN):

$$Precision=\frac{TP}{TP+FP}\ast 100\%,$$

(21)

$$Recall=\frac{TP}{TP+FN}\ast 100\%,$$

(22)

$$F_1=2\ast \frac{Precision\times Recall}{Precision+Recall}\ast 100\%,$$

(23)

where $F_1$ is obtained by directly averaging the F1 scores of all types, that is, Macro-F1 [45].

4.1.3. Baseline Methods

Aiming to measure and analyze the proposed method, we compare its performance on the four datasets mentioned above with mainstream models in recent years:

• Methods without lexicon features: BiLSTM-CRF [16], BERT [19], BERT-CRF [23], BERT-BiLSTM-CRF [46], BERT-GlobalPointer [37] and BERT-Pointer [36]. These models are based on deep learning, such as BiLSTM or BERT, using sequence labeling or pointer labeling frameworks. In particular, these models do not introduce Chinese lexicon-level features.
• Methods with lexicon features: lattice LSTM [24] and lexicon-enhanced BERT (LEBERT) [28]. Lattice LSTM can be seen as an extension of the character-based NER model, which adds words as input and additional gates to control information flow; LEBERT directly integrates external lexicon knowledge into the BERT layer through the lexicon adapter layer.

4.1.4. Training

In all experiments, we used the same BERT pre-trained model. Due to differences between datasets, we adjusted batch size and max_seq_len separately for each dataset, and all experiments were conducted on a single NVIDIA (Santa Clara, CA, USA) GeForce RTX 3070Ti GPU. For specific details and settings of other hyperparameters, please refer to

Table A1. Some common hyperparameter settings about the experiments.

Hyperparameter	Value
Data Augmentation Ratio	0.09
Epoch	20
BERT Model	bert-base-chinese
BERT LR	1 × 10−5
BERT Dropout Rate	0.35
Linear LR	3 × 10−4
Optimizer	AdamW
Boundary Embedding Dim	16
Multi-head self-attention head	8
Threshold	0.6

,

Table A2. Some hyperparameter settings about the experiments on the four datasets.

Dataset	max_seq_len	Batch Size
Resume	64	64
Youku	64	64
SubDuIE	256	32
PPOST	256	32

and

Table A3. Some hardware and software environments about the experiments.

Environment	Value
Processor	12th Gen Intel(R) Core(TM) i5-12600KF
RAM	32.0 GB
GPU	NVIDIA GeForce RTX 3070Ti GPU 8GB
Python Version	3.8.16
PyTorch Version	2.0.0

in Appendix A.

4.2. Results and Analysis

4.2.1. Comparative with Other Models

To validate the effectiveness of the proposed method in the Chinese NER, we applied an early stopping mechanism and calculated the average results over five runs of experiments.

Table 3. Precision, recall, and F1-score statistics on the Resume and Youku datasets.

Model	Resume			Youku
	P(%)	R(%)	F1(%)	P(%)	R(%)	F1(%)
BiLSTM-CRF	93.70	93.30	93.50 [47]	80.31	79.22	79.76
BERT	94.20	95.80	95.00 [19]	85.06	76.75	80.69
BERT-CRF	-	-	96.87 [48]	83.00	81.70	82.40 [43]
BERT-BiLSTM-CRF	-	-	95.59 [49]	89.56	80.48	84.78
Lattice LSTM	94.81	94.11	94.46 [24]	84.43	81.28	82.82
LEBERT	-	-	96.08 [28]	91.93	82.10	86.74
BERT-GlobalPointer	96.94	93.71	95.30	92.76	82.02	87.06
BERT-Pointer	97.15	94.27	95.69 [27]	95.06	83.47	88.89
BERT-LPN	98.12	95.70	96.90	95.74	83.23	89.05

Significant bold values are the current SOTA in performance achieved, and the bold in the last row of these tables denotes the new achievements we obtained in our model.

and

Table 4. Precision, recall, and F1-score statistics on the SubDuIE and PPOST datasets.

Model	SubDuIE			PPOST
	P(%)	R(%)	F1(%)	P(%)	R(%)	F1(%)
BiLSTM-CRF	72.37	58.71	64.83	61.14	47.37	53.38
BERT	79.48	55.49	65.35	66.79	46.88	55.09
BERT-CRF	78.60	57.66	66.52	68.82	46.40	55.43
BERT-BiLSTM-CRF	76.65	56.71	65.19	67.78	46.22	54.96
Lattice LSTM	79.00	62.68	69.90	71.07	50.38	58.96
LEBERT	83.31	61.58	70.81	79.48	53.13	63.69
BERT-GlobalPointer	83.20	61.84	70.95	80.40	67.98	73.67
BERT-Pointer	83.67	61.81	71.10	81.50	71.60	76.23
BERT-LPN	84.47	63.30	72.37	83.50	74.07	78.51

present the experimental results for the models mentioned in Section 4.1.3.

Experimental results on two shorter-text datasets, Resume and Youku, and two longer-text datasets, SubDuIE and PPOST, show that our model improves F1-score by 0.03%, 0.16%, 1.27%, and 2.28%, respectively, compared to previous models. This demonstrates that our feature fusion strategy effectively utilizes Chinese lexicon-level information, enhances Chinese NER performance. However, the advantages of our method are not fully reflected in the Resume and Youku datasets, where the sample texts are shorter, and entities are simpler. Moreover, the DTFW-lookup Table for the original corpus of these datasets was not calculated. The more significant improvement on longer-text datasets may be attributed to the use of the WordPiece algorithm, simplifying the model input and integrating Chinese lexicon-level features, enhancing the model’s generalization performance and feature capture capability. Specifically, on the PPOST dataset, compared to sequence labeling models, the performance of pointer labeling models has significantly improved. This experimental result indicates that pointer labeling models can effectively improve the recognition performance of long and nested entities in the domain of Chinese paleontology popular science.

4.2.2. Comparative with Various Entity Types

To comprehensively evaluate our model’s recognition performance on various entities in our domain, we collected the performance of our model in identifying different entities on the PPOST dataset, as shown in Figure 8. The experimental results indicate that the recognition accuracy of various entities is mainly positively correlated with the number of labels in the dataset. This may be due to the scarcity of samples for certain types, leading to the model’s inability to learn richer features of these type entities. Specifically, since we define the Feature type mostly for the attributes of things, such as the volume of fossils or the appearance of ancient organisms, and with diverse characteristics, the recognition performance for this part of entities is relatively lower. Therefore, it is more appropriate to consider conducting attribute extraction tasks for these entities.

To evaluate the recognition performance of our model for long entities and nested entities, we sampled 324 instances of long entities with an entity length exceeding 8 characters and an average length of 10 characters and 213 instances of nested entities from the PPOST dataset. The relevant experimental results are shown in

Table 5. Precision, recall, and F1-score statistics of long entities and nested entities on the PPOST.

Model	Long Entities			Nested Entities
	P(%)	R(%)	F1(%)	P(%)	R(%)	F1(%)
BiLSTM-CRF	43.36	39.76	41.48	47.68	40.01	43.50
BERT	46.90	41.47	44.02	45.19	37.31	40.87
BERT-CRF	48.43	41.29	44.58	42.17	35.31	38.44
BERT-BiLSTM-CRF	49.00	40.39	44.28	43.16	37.99	40.41
Lattice LSTM	51.49	45.62	48.38	44.11	41.01	42.50
LEBERT	52.04	45.84	48.74	45.80	42.61	44.15
BERT-GlobalPointer	57.42	53.73	55.51	65.80	56.76	60.95
BERT-Pointer	59.00	54.71	56.77	67.67	58.30	62.64
BERT-LPN	60.39	55.19	57.67	68.32	59.26	63.47

. The results indicate that compared to models based on CRF, models based on pointer labeling can effectively enhance the recognition performance of long and nested entities. Moreover, our BERT-LPN model achieves the best performance among the models mentioned.

4.2.3. Comparative with Complexity of Various Models

Compared to neural networks, CRF models themselves have a relatively high time performance overhead [50], with an overall time complexity of $O(m\ast n^2)$, where $m$ represents the length of the observed sequence, and $n$ denotes the number of hidden states. To assess the computational requirements of our model, we only recorded the trainable parameters and training speed of the pointer labeling models, as shown in

Table 6. Parameters and training speed statistics of the GlobalPointer, LPN, and Pointer models based on BERT on the PPOST dataset.

Model	Parameters	Speed
BERT-GlobalPointer	29.00 M	1× *
BERT-LPN	28.97 M	1.15× *
BERT-Pointer	28.94 M	1.17× *

* 1.15× and 1.17× represent 15% and 17% faster than 1×, respectively.

. The experimental results indicate that the additional time performance overhead of our method is negligible. This is because, through preprocessing, we obtained the DTFW-lookup Table, eliminating the need for repetitive computation of lexicon frequency weight during training, requiring only table lookup operations. Additionally, compared to the BERT-GlobalPointer model, we used only two fully connected layers for positional marking, reducing the parameters of fully connected layers and improving the training speed by 15%. In contrast to the BERT-Pointer model, we need to calculate the boundaries of lexicon and look up the DTFW-Lookup Table to obtain lexicon frequency weights. Moreover, the boundary embedding layer is trainable, resulting in an additional 1.7% performance overhead.

4.2.4. Comparative with Various Strategies

In order to investigate the contribution of each strategy to our approach, we conducted a series of ablation experiments on the SubDuIE and PPOST datasets, and the results are shown in Figure 9.

The ablation studies were designed as follows:

• Strategy 1: Strategy of adding only domain term frequency weight features without additional methods for feature fusion;
• Strategy 2: Strategy of adding only Chinese lexicon boundary features without additional methods for feature fusion;
• Strategy 3: Strategy of simultaneously adding Chinese lexicon boundary and domain term frequency weight features without additional methods for feature fusion;
• Strategy 4: Strategy of simultaneously adding Chinese lexicon boundary and domain term frequency weight features, using the BiLSTM for feature fusion.

We remove any strategy that will lead to a decrease in entity recognition accuracy. Specifically, compared to the original pointer network, Strategy 1, Strategy 2, and Strategy 3 have all shown improvements on our experimental SubDuIE and PPOST datasets. Their F1-scores increased by 0.28%, 0.54%, 0.67% and 0.67%, 0.72%, 1.84%, respectively. This indicates that lexicon boundary features and domain term frequency weight features can effectively integrate Chinese lexicon-level features, improving the performance of the NER model. Compared to Strategy 3, Strategy 4 and the Complete model showed F1-score improvements of 0.13% and 0.44% and 0.18% and 0.44%, respectively, on the above two datasets. This suggests that the feature fusion of Chinese lexicon-level feature vectors can more effectively integrate contextual information features, enhancing the performance of the NER model. Compared to Strategy 4, the Complete model showed F1-score improvements of 0.31% and 0.26% on the above two datasets, respectively, indicating that the multi-head self-attention mechanism better handles dependencies between information.

4.2.5. Comparative with Various Data Augmentation Ratios

To verify the effectiveness of the data augmentation method, we conducted an additional set of experiments on the BERT-LPN model. The experiments were performed on the PPOST dataset, and the results are shown in Figure 10. The experimental results indicate that, on the PPOST dataset, our proposed data augmentation method has a certain performance improvement within the enhancement ratio range of (0, 0.1], where the ratio value is the ratio of the generated data to the original total data. This improvement may be attributed to the model further learning the ability to identify entities that do not exist in the input within this enhancement ratio range. However, when the ratio is more than 0.1, the model’s performance sharply declines, and the performance is even worse than that of the original dataset. This may be due to an excess of augmented data in the training set, causing the model to preferentially predict the absence of entities in the input.

4.3. Extraction Results and Error Analysis

In this study, a set of experiments were conducted to validate the extracted NER results and analyze errors. Some examples of experimental results are shown in

Table 7. Illustrative examples of recognition results.

ID	Sentence	Ground Truth	Prediction
1	“寒武纪地层记录” (Record of Cambrian strata)	寒武纪 (Cambrian), 寒武纪地层 (Strata).	寒武纪 (Cambrian) √, 寒武纪地层 (Strata) √.
2	“中国科学院古脊椎动物与古人类研究所” (Institute of Vertebrate Paleontology and Paleoanthropology, Chinese Academy of Sciences)	中国科学院 (Chinese Academy of Sciences), 古脊椎动物与古人类研究所 (Institute of Vertebrate Paleontology and Paleoanthropology).	中国科学院 (Chinese Academy of Sciences) √, 古脊椎动物与古人类研究所 (Institute of Vertebrate Paleontology and Paleoanthropology)√.
3	“主要分布于我国湖南, 江西、浙江、安徽等地” (Mainly distributed in Hunan, Jiangxi, Zhejiang, Anhui and other places in China)	湖南 (Hunan), 江西 (Jiangxi), 浙江 (Zhejiang), 安徽 (Anhui).	湖南 (Hunan) √, 江西 (Jiangxi) √, 浙江 (Zhejiang) √, 安徽 (Anhui) √.
4	“恐龙位于爬行动物和鸟类之间的十字路口” (Dinosaurs were at the crossroads between reptiles and birds)	恐龙 (Dinosaurs ), 爬行动物 (Reptiles), 鸟类 (Birds).	恐龙 (Dinosaurs ) √, 爬行动物 (Reptiles) √, 鸟类 (Birds) √.
5	“我国华南地区中—上奥陶统黑色页岩” (Middle-Upper Ordovician black shale in South China)	华南地区 (South China), 中—上奥陶统 (Middle-Upper Ordovician).	华南地区 (South China) √, 上奥陶统 (Upper Ordovician)×.
6	“獬豸盘角鹿属于长颈鹿科” (Discokeryx xiezhi belongs to the family Giraffeidae)	獬豸盘角鹿 (Discokeryx xiezhi), 长颈鹿科 (Giraffeidae).	盘角鹿 (Discokeryx) ×, 长颈鹿科 (Giraffeidae) √.

“^×” represents entity type or boundaries recognition errors. “^√” represents complete and accurate recognition of entity type and boundaries.

. From the experimental data with IDs 1 and 2, it can be observed that our model can effectively recognize nested entities, such as “寒武纪地层” (Cambrian strata), and long entities, like “古脊椎动物与古人类研究所” (Institute of Vertebrate Paleontology and Paleoanthropology). From the experimental data with IDs 3 and 4, it is evident that when the input data contains multiple entities of the same type (Address: “湖南” (Hunan), “江西” (Jiangxi), “浙江” (Zhejiang), “安徽” (Anhui)), our model, with the introduction of relative positional information, can effectively distinguish the boundaries of these four entities, reducing the likelihood of erroneously combining arbitrary start and end positions to form incorrect entities.

Additionally, we summarized and analyzed the model’s recognition errors, identifying the following main issues:

• We cannot accurately identify some continuous entity characters separated by the symbol “—”. For example, experimental data with ID 5 indicates that in the entity “中—上奥陶统” (Middle—Upper Ordovician), “中—” (Middle—) is not considered part of the entity. This may be due to a limited number of samples for such entities or the negative impact of the “—” symbol on the recognition of contextual features;
• We cannot accurately identify entities with uncommon terms as boundaries. For instance, experimental data with ID 6 shows that in the entity “獬豸盘角鹿” (Discokeryx xiezhi), the model fails to recognize “獬豸” (xiezhi) as part of the entity. This could be because “獬豸” (xiezhi) is an uncommon term, and its domain term frequency weight is relatively low, preventing the model from fully learning the features of uncommon lexicons.

5. Conclusions and Future Work

In the domain of Chinese paleontology popular science, NER is a fundamental step for extracting information and knowledge from a massive amount of popular science articles or books. In this study, we researched and compared mainstream Chinese deep learning-based NER methods to address the challenges in our domain. We constructed a corpus for the Chinese paleontology popular science domain and its annotated NER dataset, PPOST. We proposed a domain NER model, BERT-LPN, based on the BERT pre-trained language model. The model was applied to four different datasets, and its performance was evaluated in terms of precision, recall, and F1-score. The experimental results show that the proposed BERT-LPN method significantly outperforms baseline models and other deep learning models in both general domain and Chinese paleontology popular science texts.

The contributions of this research can be viewed from two perspectives. Methodologically, this study introduces a deep learning model named BERT-LPN. Experimental results demonstrate that, compared to other deep learning models, BERT-LPN performs better in extracting entities, providing multiple advantages for Chinese NER, including the following:

• Domain-specific information: In the task of Chinese NER within a specific domain, lexicon-level features can encompass domain-specific lexicons, thereby enhancing the model’s performance within that domain;
• Improved semantic understanding: Lexicon-level features can assist the model in better comprehending the semantic information within the text, capturing the meanings of lexicons and their contextual relationships. This helps the model in more accurately distinguishing between different entity types in longer texts.

From an application perspective, this study developed a Chinese paleontology popular science NER dataset, promoting the development of NER, popular science knowledge graphs, and popular science education in the Chinese paleontology domain. Moreover, in other NER tasks in different domains, researchers typically have easy access to original corpora, and our method relies only on the target domain’s corpus, greatly increasing the possibility of generalizing our model to other domains. Future research will focus on the following three aspects:

• Corpus construction is an ongoing process, and we will further optimize the Chinese paleontology popular science corpus and PPOST dataset. Specifically, for the Feature entity type, it will be split into more specific attributes to further improve NER performance;
• Studying the performance of our model in other Chinese domains to promote the development of Chinese NER;
• Studying how to handle ambiguous entity boundaries and uncommon terms situations based on our approach.