Entity Relationship Extraction Based on a Multi-Neural Network Cooperation Model

Abstract

Entity relation extraction mainly extracts relations from text, which is one of the important tasks of natural language processing. At present, some special fields have insufficient data; for example, agriculture, the metallurgical industry, etc. There is a lack of an effective model for entity relationship recognition under the condition of insufficient data. Inspired by this, we constructed a suitable small balanced data set and proposed a multi-neural network collaborative model (RBF, Roberta–Bidirectional Gated Recurrent Unit–Fully Connected). In addition, we also optimized the proposed model. This model uses the Roberta model as the coding layer, which is used to extract the word-level features of the text. This model uses BiGRU (Bidirectional Gated Recurrent Unit)–FC (Fully Connected) as the decoding layer, which is used to obtain the optimal relationship of the text. To further improve the effect, the input layer is optimized by feature fusion, and the learning rate is optimized by the cosine annealing algorithm. The experimental results show that, using the small balanced data set, the F1 value of the RBF model proposed in the paper is 25.9% higher than the traditional Word2vec–BiGRU–FC model. It is 18.6% higher than the recent Bert–BiLSTM (Bidirectional Long Short Term Memory)–FC model. The experimental results show that our model is effective.

1. Introduction

In the current era of big data, all kinds of data and information are growing explosively [1], and artificial intelligence technology is developing rapidly with the support of sufficient data. However, the problem of how to make machines think like people has not been solved, there is still a gap between the development of perceptual intelligence and people’s expectations. Knowledge graph technology can provide background knowledge for machine thinking, and become an important way to realize artificial cognitive intelligence. Entity relationship extraction is one of the key links in the construction of a knowledge graph.

Compared with English entity relationship extraction, Chinese entity relationship extraction has the characteristics of a relative lack of open research datasets and a more prominent importance of word-level features [2]. In addition, how to ensure a better effect of the model under the condition of a small dataset has always been a major challenge in the field of entity relationship extraction [3]. To solve the above problems, we propose a multi-neural network collaborative RBF model, which can achieve good results under the condition of small balanced samples. The multi-neural network cooperation model proposed in this paper is a new combination of multiple deep learning models. This paper also proposes an optimization method for the multi-neural network cooperation model.

The novel contributions of our work are summarized as follows:

(1) Proposing the idea of conducting a relationship extraction experiment under the condition of a balanced small sample. This paper constructs a small, balanced sample relationship extraction dataset and can judge which model is better.

(2) Aiming at the task of entity relationship extraction under the condition of a small balanced dataset, we studied several deep learning models and the effect of multi-model collaboration. Given this, this paper put forward a new multi-neural network cooperation model, namely the RBF (Roberta–BiGRU–FC) model, based on our research. The Roberta–BiGRU–FC model uses Roberta as the coding layer and BiGRU–FC as the decoding layer. The F1 value of the experimental result on the constructed small sample data set reached 89.6%.

(3) Proposing an effective optimization method for the RBF multi-neural network cooperation model. We optimized the RBF model on the input feature layer and learning rate to improve the effect on the task of extracting entity relations from small samples. The learning rate of the entity relationship extraction model is generally fixed, and it cannot be optimized adaptively according to the transformation of the loss function. The cosine annealing algorithm can make the learning rate dynamically adjusted, and the learning rate first slows down and then accelerates to decline. Given this, this paper applied the cosine annealing algorithm to the optimization of RBF models. After optimization, the F1 value on the constructed small sample dataset reached 91.9%.

The rest of this paper is organized as follows: the second section briefly introduces the relevant work, the third section introduces the overall architecture of the RBF model, the fourth section describes the process and results of the experiment, the fifth chapter analyzes the experimental results, and the sixth section summarizes the content of the paper and describes the next steps.

2. Related Work

How to extract the required triples (head-entity, relationship, tail-entity) from the current massive unstructured text and form a knowledge base has always been the focus of research in the field of knowledge graphs [4]. At present, the mainstream methods of triplet extraction include pipeline extraction and joint extraction. Pipeline extraction takes entity recognition and relationship extraction as different tasks and models in series. Joint extraction takes entity recognition and relationship extraction as one task. In contrast, joint extraction modeling is more complex and lacks flexibility compared with the pipeline method [5,6], so this paper mainly explores entity relationship recognition in pipeline extraction.

Methods for entity relation extraction can be summarized into three categories: template rules-based methods, traditional machine learning-based methods, and deep learning-based methods [7,8,9].

2.1. Entity Relationship Extraction Based on Template Rules

The method based on template rules was mainly intended to find the characteristics and laws of entity relations when they appear in the text and set a series of template rules by using regular expressions [10]. In 1998, Aone et al. proposed that experts design template rules [11], detect and match the text, and match the entity relations that conform to the template. Later, there were also methods such as using a syntactic analyzer to construct text dependencies [12] to improve the efficiency of building template rules. The method based on template rules often needs the experienced support of specific fields in specifying rules, and the rule templates in different fields often cannot be used mutually, which has characteristics of poor universality. It is difficult to use the same set of rules effectively across fields [13].

2.2. Entity Relation Extraction Based on Traditional Machine Learning

Most traditional machine learning models mainly focus on extracting features from sample data. These models usually use classical models such as SVM (Support Vector Machine), the Markov chain, and logistic regression [14]. In 2005, Zhou et al. [15] used the SVM machine learning model for entity relationship extraction, and the F1 value on the ACE database reached 55.5%. In 2006, Culotta, A. and others applied the Markov chain model to the task of entity relationship extraction [16]. In 2009, Li [17] and others processed the data input layer of SVM and made some achievements in relationship extraction in the biomedical field by using word bag features, part-of-speech features, and dependency features as model inputs.

Due to the defects of traditional machine learning-based entity relationship extraction methods and the propagation of feature extraction errors, it is often difficult to obtain a good recall rate in small sample entity relationship extraction.

2.3. Entity Relationship Extraction Based on Deep Learning

With the rise of deep learning technology, more and more deep learning models, such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM, and GRU, are used in entity relationship extraction tasks [18,19,20,21].

In 2013, Liu et al. first proposed using the CNN method for relationship classification and encoding the input words through the dictionary of synonyms. The model mainly included a convolution layer, full connection layer, and softmax classification [22], which inspired the subsequent deep learning model for relationship extraction. In 2014, Zeng et al. proposed using word vectors and position vectors as the input of a convolution neural network, which could better improve the effect of relationship extraction [19]. In 2015, Zhang et al. [23] proposed using RNN for relationship extraction, without using any lexical features; the effect was similar to that of CNN combined with lexical features. In 2016, Zhou et al. [24] proposed LSTM (long short-term memory) instead of the RNN model, added an attention mechanism, proposed BiLSTM attention model architecture, and used word vectors and position vectors as feature representations of the model input layer, which improved the effect of relationship recognition.

Most of the above studies used the traditional pre-training method [25] to vectorize the text. Among them, the Word2vec text feature extraction model [26] released by Google in 2013 had a good effect in the traditional method, but Word2vec is a static method and cannot be dynamically optimized for specific tasks.

In 2018, Google released Bert’s pre-training model [27], which could obtain the dependence of dynamically encoded word vectors to capture longer distances by fully extracting words and sentences [28,29]. At the end of October 2018, it announced Bert’s performance in 11 NLP (Natural Language Processing) tasks, and Bert achieved good results [30]. Zhuang et al. proposed the Roberta model in 2021 [31]. The Roberta model added batch size and training data (more than 100 GB) based on the Bert model and used double-byte coding in language representation [32], which improved the accuracy of vocabulary representation and task execution efficiency.

Therefore, based on the above comparative analysis, deep learning models perform well in relation extraction tasks, and we propose a new method of multi-neural network cooperation for relation extraction based on deep learning. The proposed Roberta–BiGRU–FC multi-neural network cooperation model uses Roberta’s pre-training model as an encoder and BiGRU and FC model as a decoder, and the model was optimized to better extract entity relations under the condition of a small, balanced dataset.

3. Proposed Method

Aiming at the problem of entity relationship recognition in a small dataset, the overall framework of the RBF model proposed in this paper is illustrated in Figure 1. It mainly includes four levels: (1) the text layer, which refers to the preprocessed text; (2) Roberta layer, using the Roberta pre-training model for text feature coding; (3) the BiGRU layer, decoding text features; and (4) the FC layer, mapping the result to the corresponding relationship category, and outputting the final relationship value according to the rules.

3.1. Roberta Layer

In the coding layer, in addition to the text that needed to be processed originally, we also inputted the entity pairs as external features and performed feature fusion in the tokenizer stage to improve the effectiveness of the final model. As illustrated in Figure 2, text information was first inputted into Tokenizer Dictionaries, which obtained token vectors based on dictionary-matching methods.

The token embeddings obtained above only represent the characteristic value of a single word at the word level. In addition, other features also have a significant impact on the entity relationships to be extracted, such as text sorting and inter-line order. Therefore, as illustrated in Figure 3, in feature processing it is necessary to integrate features such as position and text line order with the characteristics of the text itself.

The original text is represented by x = {x₁, x₂, …, x_n}. After feature extraction, the calculation methods for the three features were as follows, where W represents the maximum number of characters. In this paper, the threshold of W was set to 128. If each line of text was more than 128 it would be truncated, and if it was less than 128 it would be supplemented with 0. V_k represents the dimension of a character vector.

$${\mathrm{X}}_{\mathrm{Token}}={\mathrm{R}}_1^{\mathrm{w}\times {\mathrm{V}}_{\mathrm{k}}}$$

(1)

$${\mathrm{X}}_{\mathrm{Position}}={\mathrm{R}}_2^{\mathrm{w}\times {\mathrm{V}}_{\mathrm{k}}}$$

(2)

$${\mathrm{X}}_{\mathrm{Segment}}={\mathrm{R}}_3^{\mathrm{w}\times {\mathrm{V}}_{\mathrm{k}}}$$

(3)

The input of the transformer layer is E = X_Token + X_Position + X_Segment, where CLS (classification) and SEP (separator) fields are separators specified by Roberta.

After embedding inputs into the Roberta model, the Roberta model continuously extracts sentence features through the multi-head attention mechanism method. Firstly, it multiplies the random initialization matrix M with the fusion feature vector of the above input to obtain vectors Q, vectors K, and vectors V, and then updates and optimizes the values of these three vectors as the training process proceeds. The attention mechanism function represents a mapping from Q to a series of K, and then to V, The expression formula is as follows, and d_k represents the dimension of Q.

$$\mathrm{Attention}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=\mathrm{softmax}(\frac{{\mathrm{QK}}^{\mathrm{T}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}})\mathrm{V}$$

(4)

To prevent the effect of an iteration from being bad, the model performs residual connection and layer normalization after multi-head attention calculation and converts the input vector into data with a mean value of 0 and a variance of 1. The normalization formula is as follows, where E[x] is the mean value of the sample and σ²[x] is the variance of the sample:

$$\mathrm{y}=\frac{{\mathrm{x}}_{\mathrm{i}}-\mathrm{E}\left[\mathrm{x}\right]}{\sqrt{{\mathsf{\sigma }}^2+\mathsf{\epsilon }}}\ast \mathrm{r}+\mathsf{\beta }$$

(5)

The whole process needs to be iterated many times without a segment optimization effect. The structure of the network in a single iteration is illustrated in Figure 4.

3.2. BiGRU Layer

The GRU network is a variant of the LSTM network. It is a variant model of recurrent neural networks that can effectively handle sequential data of text line types. The GRU network converts the forgetting gate and input gate in the LSTM network into an update gate to solve the problem of ladder explosion caused by too long text [33]. The GRU model is simpler and more efficient than the standard LSTM model. Because of this, this paper used the BIGRU model to further decode text feature vectors, hoping to obtain better entity-relationship data features.

This layered network mainly included an update gate and a reset gate. The update gate mainly controlled the retention or deletion of some information from the forward state, and the recharge gate was used to control whether the calculation of the candidate state was dependent on the previous state.

The calculation formula for the updated door is as follows:

$${\mathrm{z}}_{\mathrm{t}}=\mathsf{\sigma }\left({\mathrm{W}}_{\mathrm{Z}}·{\mathrm{I}}_{\mathrm{t}}+{\mathrm{U}}_{\mathrm{Z}}{\mathrm{P}}_{\mathrm{t}-1}\right)$$

(6)

The calculation formula for reset door is as follows:

$${\mathrm{r}}_{\mathrm{t}}=\mathrm{S}\left({\mathrm{W}}_{\mathrm{t}}·{\mathrm{I}}_{\mathrm{t}}+{\mathrm{U}}_{\mathrm{t}}{\mathrm{P}}_{\mathrm{t}-1}\right)$$

(7)

The calculation formula of the implicit state is as follows:

$$\widetilde{{\mathrm{P}}_{\mathrm{t}}}=\tan \mathrm{h}\left(\mathrm{W}·{\mathrm{I}}_{\mathrm{t}}+\mathrm{U}\left({\mathrm{r}}_{\mathrm{t}}\otimes {\mathrm{P}}_{\mathrm{t}-1}\right)\right)$$

(8)

The calculation formula of the output state is as follows:

$${\mathrm{P}}_{\mathrm{t}}=\left(1-{\mathrm{z}}_{\mathrm{t}}\right)\otimes {\mathrm{P}}_{\mathrm{t}-1}+{\mathrm{z}}_{\mathrm{t}}\otimes \widetilde{{\mathrm{P}}_{\mathrm{t}}}$$

(9)

I_t represents the features entering the model, h_t−1 represents the characteristics of the input at time t − 1, and h_t represents the characteristics of the input at time t. S is a sigmoid function, and the value range of S is from 0 to 1.

3.3. Full Connection Layer

At the end of the model, this paper used the FC layer to perform dimensional changes on the output of the model and match the correct size so that the output dimension of the whole network was equal to the number of categories of entity relationship classification, to obtain the optimal entity relationship category. The formula used in the full connection layer is illustrated in Formula (10): m represents the input matrix, A represents the weight matrix, and B represents the paranoid matrix.

$$\mathrm{y}={\mathrm{mA}}^{\mathrm{T}}+\mathrm{b}$$

(10)

3.4. Loss Function

The loss function was used to calculate the difference between the predicted result and the real value. The model should use the cross-entropy function to determine the proximity between the actual output and the expected output. Its calculation formula is as follows, where p is the result of the model output and q is the label corresponding to the data.

$$\mathrm{loss}\left(\mathrm{p},\mathrm{q}\right)=-\log (\frac{\exp \left(\mathrm{p}\left[\mathrm{q}\right]\right)}{{\sum }_{\mathrm{j}}\exp \left(\mathrm{p}\left[\mathrm{q}\right]\right)})$$

(11)

3.5. Learning Rate Optimization Based on Cosine Annealing Algorithm

In deep learning technology, the gradient descent algorithm is generally used to optimize the objective function during the operation of the model. When the loss function is at the minimum value, the learning rate should be adjusted to minimize it, to prevent excessive parameter update and overfitting. Especially under the condition of a small balanced sample, the effect is better and more obvious.

The current mainstream methods include the Genetic Algorithm, Particle Swarm Optimization, the Differential Evolution algorithm, the Exponential Decay algorithm, and the Cosine Annealing algorithm. The Genetic Algorithm, Particle Swarm Optimization, Differential Evolution algorithm, and Exponential Decay algorithm are prone to premature convergence and poor convergence ability for high-dimensional complex problems of multi-neural network cooperation [34,35], especially the Particle Swarm Optimization algorithm, which is very prone to premature convergence [36]. Therefore, this paper selected the Cosine Annealing algorithm [37] to optimize the learning rate based on the fusion optimization of the feature layer, and the effectiveness of the method for the results was verified by experiments.

Based on this, the cosine annealing algorithm was adopted to make the learning rate change according to the law of the cosine cycle, In a cosine cycle, the learning rate slows down first and accelerates the decline. In each cycle, the cosine annealing of the learning rate is carried out according to the following formula, where lr_max is the highest learning rate initially set, lr_min is the lowest learning rate initially set, T_cur is the number of iterations epochs since the last restart, and T is the total number of iterations epochs contained in a cycle.

$$\mathrm{lr}={\mathrm{lr}}_{\min }+\frac{1}{2}\left({\mathrm{lr}}_{\max }+{\mathrm{lr}}_{\min }\right)\left(1+\cos \left(\frac{{\mathrm{T}}_{\mathrm{cur}}}{\mathrm{T}}\right)\mathsf{\pi }\right)$$

(12)

This model is the optimal multi-neural network cooperation model found through a large number of experimental processes for small sample conditions. The main differences from existing models are reflected in two aspects: firstly, the various layers in the multi-neural network are different, similar to the existing BBF (BERT–BiGRU–FC) model, but the experimental effect of RBF is stronger than the BBF model. Secondly, there are differences in the optimization methods used. This paper innovatively uses the cosine annealing algorithm to optimize the multi-neural network cooperation model, and the experimental results show that the optimization effect is effective.

4. Experiments and Results

4.1. Construction of the Small Sample Dataset

DuIE2.0 is a commonly used public dataset for entity relationship extraction at present. Its triple composition mainly comes from Baidu Encyclopedia, Baidu feed flow text, and Baidu tieba. It contains a total of 50 entity relationships, as well as up to 172,983 training sets and 19,981 testing sets. However, there is no average sampling for different relationship categories in DuIE2.0, and there are few small samples of public datasets about entity relationship extraction [38].

To construct the experimental dataset, we analyzed the published DuIE2.0 entity relationship dataset for optimization and simplification. On this basis, we proposed a method that was used to construct a small and balanced dataset. It mainly sets the average value of relationships for big datasets and performs average sampling on the relationships within the big dataset to obtain a simplified average sampling for small datasets. To construct an average sampling small sample dataset, during the experimental process this paper simplified the setting of an average sampling upper limit of 30 for the training set relationship and 10 for the testing set average sampling upper limit. Finally, we built a small sample data set with 462 rows of test data, 1451 rows of training data, and 50 relationship types (including an unknown relationship). Although the dataset was small, there were many types of relationships; this also increased the difficulty of relationship extraction. Some examples of sampling results are illustrated in

Table 1. Example of dataset sampling results.

Relationship Number	DuIE2.0 Train Set	Our Small Balanced Train Set	DuIE2.0 Test Set	Our Small Balanced Test Set
1	1817	30	171	10
2	4701	30	460	10
3	2466	30	237	10
4	16,553	30	1425	10
5	10,342	30	889	10
6	660	30	50	10
7	1008	30	94	10
8	3522	30	337	10
…	…	…	…	…
17	20	20	1	1
…	…	…	…	…
50	633	30	57	10
Total number	172,983	1451	19,981	462

.

4.2. Evaluation Index

For the evaluation of experimental results, this paper takes the F1 value as the main evaluation index based on current mainstream practices. P is the precision rate of experimental results, R is the recall rate of experimental results, FP is the number of times predicted to be the relationship but not the relationship, TP is the number of correct predictions of the relationship, FN is the number of times predicted to be the relationship but wrong, and the specific calculation formula of F1 value is as follows:

$$\mathrm{P}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(13)

$$\mathrm{R}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(14)

$$\mathrm{F}1=\frac{2\mathrm{P}\ast \mathrm{R}}{\mathrm{P}+\mathrm{R}}$$

(15)

4.3. Baselines

In terms of multi-neural network cooperation for the relation extraction task, we compared our model with the following models:

• In 2015, Shu Zhang, etc., proposed using bidirectional long short-term memory networks for relation classification [39].
• In 2018, Zhao Ming proposed a Word2vec–BIGRU–FC model for relationship extraction [40].
• In 2020, Zihan Wang, etc., proposed a Bert–BiLSTM–FC model for relationship extraction [41].
• In 2021, YUE Qi and Li Xiang proposed a Bert–RNN–FC model for relationship extraction [42].
• In 2022, Gupta, etc., proposed a Bert–BiGRU–FC model for relationship extraction [43].

Because there are few existing models for relation extraction under the condition of Chinese small samples, and we were unable to find more mature baseline models, this paper also used the variant of the RBF model (before optimization) as a baseline and the variant of the RBF model using the Exponential Decay algorithm as a baseline.

4.4. Parameter Setting

The server configuration used in this experiment was Intel (R) Core (TM) i7-10875H (2.3 GHz), with 32 GB memory, NVIDIA Geforce RTX 2060 GPU, and a Windows 11 64-bit operating system. The version of the PyTorch framework the experiment used was 1.7.1. The version of Word2vec used was Tencent-ailab-embedding-zh-d100-v0.2.0-s from Tencent AILab. The experiment also used the transformers model, whose version number was 2.5.1, and the coding layer used the Roberta model and the BiGRU model.

The main parameter settings of the Roberta are illustrated in

Table 2. The parameter settings of the Roberta model.

Parameter	Value
hidden_size	768
max_position_embeddings	512
num_attention_heads	12
num_hidden_layers	12
pooler_fc_size	768
pooler_num_attention_heads	12
pooler_num_fc_layers	3
pooler_size_per_head	128
vocab_size	21,128

.

The main parameter settings of the BiGRU model are illustrated in

Table 3. The BiGRU model parameter settings.

Parameter	Value
input_size	768
hidden_size	64
dropout	0.4

.

The learning rate was optimized based on cosine annealing. According to Formula (11), the initial maximum learning rate was set to 0.01 and the cycle period of the learning rate was set to 5 epochs. The optimization of the learning rate in our method is illustrated in Figure 5.

4.5. Main Results and Analysis

In terms of experimental comparison, on the small sample corpus, this method was compared with two types of current commonly used entity classification methods. In the feature extraction layer, experimental comparisons were conducted on the traditional Word2vec model and Bert model, and the Word2vec model used version 0.2.0 “Tencent AI Lab embedding Corps” published by Tencent AI laboratory. The Bert model used the Roberta model. The parameter settings and results obtained are illustrated in

Table 4. Comparison of experimental results.

No.	Model	Training Set	Precision	Recall	F1	Best Epoch
1	Word2vec–GRU–FC	Epoch = 145, learning_rate = 0.002, embedding_dim = 100, hidden_dim = 64, BATCH = 4, dropout = 0.4	44.4%	41.5%	42.6%	138
2	Word2vec–BiGRU–FC (Zhao Ming et al., 2018 [40])		65.4%	66.6%	66.0%	46
3	Word2vec–LSTM–FC		23.6%	32.9%	27.5%	139
4	Word2vec–BiLSTM–FC (Shu Zhang et al., 2015 [39])		58.1%	59.2%	58.6%	131
5	Bert–BiGRU–FC (Gupta et al., 2022 [43])		75.6%	74.4%	76.0%	134
6	Bert–BiLSTM–FC (Zihan Wang and Bo Yang, 2020 [41])		74.6%	72.0%	73.3%	138
7	Bert–RNN–FC (YUE Qi and Li Xiang, 2021 [42])		73.5%	72.4%	72.9%	89
8	RBF before optimization	Epoch = 145, learning_rate = 0.004 dim = 100, hidden_dim = 64, BATCH = 4, dropout = 0.4	89.8%	89.4%	89.6%	34
9	RBF using Exponential Decay algorithm	Epoch = 145, learning_rate using the Exponential Decay algorithm, embedding_dim = 100, hidden_dim = 64, BATCH = 4, dropout = 0.4	90.8%	90.2%	90.5%	47
10	Our RBF model	Epoch = 145, learning_rate using cosine annealing, embedding_dim = 100, hidden_dim = 64, BATCH = 4, dropout = 0.4	91.0%	91.6%	91.9%	47

.

Due to the small amount of data, the times required for completing 145 epochs in columns 8, 9, and 10 with better training results in Table 4 were 1:31:02, 1:36:27, and 1:27:40. There was not much difference between them. The F1 and the loss value of other models on multi-neural network cooperation under different iteration times are illustrated in Figure 6 and Figure 7. They mainly included two categories: using Word2vec as the encoder, and using Bert as the encoder.

The relationship extraction effect of our RBF model before and after optimization is illustrated in Figure 8.

Further experiments were conducted to optimize the model. For the RBF model, the cosine annealing algorithm was compared with the exponential decay algorithm, and the results are shown in Figure 9.

5. Discussion

It can be seen from Table 4 that there was little difference between the precision rate, recall rate, and F1 value, indicating that there were no overfitting phenomena in this experiment, indicating the effectiveness of this experiment.

As can be seen from the Table 4 results, in the task of extracting entity relations from small samples, the precision rate of the proposed model was 91.0%, the recall rate was 91.6%, and the F1 value was 91.9%. The F1 value of the proposed model is 25.9% higher than the Word2vec–BiGRU–FC model and 33.1% higher than the Word2vec–Bilstm–FC model. For the current mainstream models, compared with Bert–BiCNN–RNN, the F1 value increased by 19%; compared with Bert–BiLSTM–FC, the F1 value increased by 18.6%; and compared with Bert–BiGRU–FC, the F1 value increased by 15.9%. It can be seen from the above results that the method proposed in this paper was greatly improved compared with other methods under the condition of small samples.

It can be seen in Figure 6 that when the number of epochs increased to 50, the F1 value tended to be stable at around 60% with Word2vec as the encoder. It can be seen in Figure 7 that when the number of epochs increased to 30, the F1 value tended to be stable around 70% with Bert as the encoder. It can be concluded that under the condition of small samples, the effect of using Bert as an encoder is better than using traditional Word2vec; this may be due to the strong fitting ability of Bert. In addition, using BiGRU as a decoder is better than using other models as decoder; it shows the excellent decoding ability of the BiGRU model, but the effect is still not ideal. Given this, we put forward the Roberta–BiGRU–FC model and optimized the feature fusion of the input layer and the learning rate of the model.

It can be seen in Figure 8 that the optimized model had a good effect in the first round of epochs. The effect of the original model increased rapidly in the early stage; this may be due to the stable learning rate of the original model, but as the number of epochs increased to 30, the effect of the optimized model became better than the original model and tended to be stable around 91%. By comparing with the exponential decay algorithm in Figure 9, the result showed that the cosine annealing algorithm could better converge the proposed multi-neural network cooperation model and obtain better results, and the optimization of the feature input layer and learning rate could improve the training effect of the RBF model; our optimization of the RBF model was effective with a small data set.

According to the above experimental results, it can be seen that, first, the small balanced dataset constructed in this paper could support the detection of the model; second, compared with other models, the RBF model proposed in this paper could achieve better results in the task of extracting entity relations from the small balanced dataset; third, we had a certain effect on the optimization of RBF model.

6. Error Analysis

From the experimental results in Table 4 and Figure 8, it can be seen that although the optimal F1 value of the RBF model was higher than other models, there was, however, a lack of stability in the image, especially after the use of the cosine annealing algorithm. There was significant shaking in the first 40 epochs, one shaking episode around the 80th epoch, and one shaking episode around the 130th epoch. Therefore, the drawback of this model is that it is prone to significant F1 value jitter during the training process.

7. Conclusions

Aiming at the task of entity relationship extraction on a small balanced sample dataset, this paper proposed a multi-neural network collaborative RBF model. The model was also optimized for learning rate by cosine annealing. In addition, this paper constructed a relatively uniformly sampled small-sample dataset. Experiments showed that compared with other models, this method could achieve good results in entity relationship extraction tasks. In subsequent research, we will conduct more comparative experiments on optimizing learning rates, such as introducing genetic algorithms into experiments and conducting comparisons, and we will make more efforts in the training effect of the model, such as introducing the attention mechanism into the decoding layer of the model and improving the performance of the model to a certain extent.