Lightweight Facial Expression Recognition Based on Class-Rebalancing Fusion Cumulative Learning

Abstract

In the research of Facial Expression Recognition (FER), the inter-class of facial expression data is not evenly distributed, the features extracted by networks are insufficient, and the FER accuracy and speed are relatively low for practical applications. Therefore, a lightweight and efficient method based on class-rebalancing fusion cumulative learning for FER is proposed in our research. A dual-branch network (Regular feature learning and Rebalancing-Cumulative learning Network, RLR-CNet) is proposed, where the RLR-CNet uses the improvement in the lightweight ShuffleNet with two branches (feature learning and class-rebalancing) based on cumulative learning, which improves the efficiency of our model recognition. Then, to enhance the generalizability of our model and pursue better recognition efficiency in real scenes, a random masking method is improved to process datasets. Finally, in order to extract local detailed features and further improve FER efficiency, a shuffle attention module (SA) is embedded in the model. The results demonstrate that the recognition accuracy of our RLR-CNet is 71.14%, 98.04%, and 87.93% on FER2013, CK+, and RAF-DB, respectively. Compared with other FER methods, our method has great recognition accuracy, and the number of parameters is only 1.02 MB, which is 17.74% lower than that in the original ShuffleNet.

1. Introduction

Facial expressions, as expressions of emotions, play a major role in interpersonal communication. FER based on lightweight networks is very important for the implementation of human–computer interaction technology. With the continuous development of FER technology, FER is popularly applied in autonomous driving, criminal investigation, medical diagnosis, psychological assessment, and auxiliary classroom teaching and other fields [1].

Recently, FER methods of deep learning based on the convolutional neural network (CNN) have gained significant achievements [2,3]. Better results have been achieved for FER on the facial expression datasets CK+ [4], JAFFE [5], and so on taken under controlled conditions (laboratory, no complex background, etc.). However, it is difficult for natural expression datasets affected by noise, lighting changes, posture, and occlusion such as FER2013 [6] and RAF-DB [7] to achieve the expected effect in FER, and scholars have studied this accordingly. Wang et al. [8] showed an attention network based on facial regions, evaluated different region generation strategies, and adaptively integrated weighted features from regions and the entire face through the attention module, which significantly improved the performance of the network under occlusion and complex pose conditions. Hamid et al. [9] showed a method to apply depth histogram metric learning to FER in CNNs, which enhanced the accuracy of FER under uncontrolled conditions. Kim et al. [10] showed a new FER framework based on a support vector machine (SVM) classifier and CNNs, which improved the ability of network feature extraction and expression classification. Gong et al. [11] showed a dual-branch multi-feature fusion network based on deep learning, in which one branch extracts multi-level facial features to enhance feature recognition capabilities and the other branch enhances the adaptability of learned features to direction and scope changes, improving the ability of network feature extraction. Although the above methods provide some effective research methods for FER in real scenes, there are still some issues: (1) Due to realistic factors such as complexity, subtlety, and occlusion of facial expression in real scenes, the robustness of the features extracted by shallow networks is poor, and it is difficult to capture the local detailed facial expressions’ features. (2) The deep network model promotes the capacity of feature extraction and the accuracy of expression recognition, but simultaneously, it greatly increases the number of network parameters and computational costs. In practical applications, limited by the cost of computing hardware configuration, the method of deepening the network depth in exchange for recognition accuracy is not practical, which is not conducive to the further development of FER.

In response to the aforementioned problems, a lightweight and efficient method of FER is shown in this paper. The following is an overview of the main innovation points of this paper:

• To resolve the issue of the uneven inter-class distribution of facial expression data, a dual-branch network (RLR-CNet) is designed in our paper.
• To alleviate the issue of easily losing feature information and to cut down on the number of parameters of the RLR-CNet, a lighter Clip_K5_ShuffleNet inverted residual structure is proposed in the RLR-CNet.
• To facilitate the transfer of facial expression feature information and promote the generalization ability of the RLR-CNet, the β-Mish activation function and the improved random masking method are used in the RLR-CNet, respectively.
• To extract facial expressions’ local key features and further enhance the accuracy of FER without significantly increasing computational complexity, a shuffle attention (SA) module is embedded into the RLR-CNet, which integrates spatial and channel attention.

2. Related Work

2.1. FER in Real-World

In the study of FER, whether through early traditional methods or popular deep learning methods, it contains three basic parts: facial recognition, feature extraction, and feature classification. In general, most methods are able to recognize facial expressions quickly and accurately in experimental situations, but the recognition performance in real scenes is greatly reduced [12,13,14], which brings great challenges to the practical application of FER.

In order to better recognize facial expressions in real scenes, more and more scholars have adopted deep learning methods such as CNN in the study of FER. Based on an existing pre-training model, Ng et al. [15] solved the problem of insufficient samples and poor characterization through multi-round fine-tuning. In the literature [16], with the complexity of the facial expression features in real scenes, a pseudo-tag generation strategy with a multi-area attention conversion network was proposed to promote the function of FER in the real scenes. Yao et al. [17] embedded a space and channel attention mechanism with HPMI in a VGG-16 network, which facilitates the flow of information between the image key information and network, and greatly solves the disappearance of facial expression feature information in real scenes. Siqueira et al. [18] designed network integration models with different structures for the datasets in the laboratory and real scenes to enhance the accuracy of FER. Because of the inter-class differentiation of facial expression data caused by various interference factors in real scenes, Shan et al. [19] proposed a method of cleverly integrating shallow features into deep features and established a local retention loss function to make the local parts in the expression class more compact and then promote the capacity of FER. Pan et al. [20,21] adopted adversarial learning methods, and guided the network to improve the recognition of expressions containing obscured faces in real scenes by combining multiple loss functions.

2.2. FER Based on Lightweight Network

Deep learning methods are highly favored in the study of FER. Deeper and more abstract expression features are extracted through larger and deeper networks or fusion attention mechanisms, and then promote the recognition accuracy. However, it also greatly increases the parameters and computational complexity of the network, reduces the recognition efficiency of the network, and affects the practical application of FER.

Therefore, in the study of FER, while improving the recognition accuracy as much as possible, it is also essential to pay attention to the equipment configuration cost of model operation in practical applications, and minimize the number of computing costs. Mahmoudi et al. [22] offered a structure that decreases the parameter quantity of the network and promotes network performance by extending the classical linear convolution function to a higher-order kernel function, which has no other weights. Kong et al. [23] utilized lightweight networks and incorporated attention mechanisms for key feature extraction, which greatly reduced the model’s computational complexity. Nan et al. [24] proposed a lightweight A-MobileNet that combines central loss and softmax loss functions to optimize the model parameters, which appreciably promotes the recognition accuracy over the original MobileNet without increasing the model parameters. Zhou et al. [25] offered a method with a multi-task cascaded network for facial recognition. To make the model more lightweight, they introduced depthwise separable convolution and residual modules into the network.

3. Models and Methods

Due to the problem of FER in real scenes, a lightweight model for FER based on class-rebalancing fusion cumulative learning is offered in this paper. Firstly, to promote the generalization ability of the network in real scenes, an improved random masking method is utilized to artificially introduce noise into the training dataset, while expanding the facial expression datasets. Then, the masked expression images are input into the dual-branch network (RLR-CNet); this network has a feature learning branch and class-rebalancing branch. For each branch, two data samples are obtained using a conventional uniform sampler and a reverse sampler, respectively, and are then fed into the corresponding branch. Under the control of cumulative learning, each branch goes through convolution and global average pooling to obtain feature vectors, which are then merged and weighted by channel fusion, and finally classified using a softmax layer. The main structure of our method is shown in Figure 1.

3.1. Dataset Construction Based on Improved Random Masking Method

In the dataset construction stage, it is a common method to promote the recognition accuracy and generalization ability of the network by optimizing the training datasets. In this research, the improved random masking method is used for randomly masking the expression images, artificially increasing the noise, expanding the training samples of the dataset, and making it closer to the real scene. Up to a point, the issue of overfitting is alleviated and the recognition efficiency of the model is enhanced [26].

Assuming the probability of masking is $p$, the area of the input facial expression image is $S=W\times H$, $S_{\mathrm{c}}$ is the area of random masking, the masking threshold is set as $S_{\mathrm{c}}/S$, the range of the masking threshold is in the interval ${(\mathrm{s}}_l,s_h)$, $r_c$ is the aspect ratio of the masking rectangle, the range of $r_c$ is in the interval $(r_1,r_2)$, and the height and width of the masking matrix area are set as $H_{\mathrm{c}}$ and $W_{\mathrm{c}}$, respectively. The specific calculation formula is as follows:

$$S_C=Rand(s_l,s_h)\times S$$

(1)

$$r_c=Rand(r_1,r_2)$$

(2)

$$H_c=\sqrt{S_c\times r_c}, W_c=\sqrt{S_c/r_c}$$

(3)

In general, a randomly selected point $P=(x_c,y_c)$ on the facial expression image, and $I_c=(x_c,y_c,x_c+W_c,y_c+H_c)$ is the area to be selected for masking. Point $P$ can be determined at any point of the target image, but considering that most of the key information of facial expression features are concentrated in the top half of the image, the coordinates of point $P$ are limited to the point at the top left 1/4 of the target image and the point at the top right 1/4 of the target image, which is recorded as $P^{\prime }=(x_{\mathrm{c}}^{\prime },y_c^{\prime })$. The masking area is updated as $I_c^{\prime }=(x_c^{\prime },y_c^{\prime },x_c^{\prime }+W_c,y_c^{\prime }+H_c)$. The process of taking the coordinates of $P$ and $P^{\prime }$ is as follows:

$$(\begin{array}{l}{x}_c=Rand(0,W)\\ y_c=Rand(0,H)\end{array}\to (\begin{array}{l}{x}_c^{\prime }=Rand(x,x_i)\\ y_c^{\prime }=Rand(y,y_j)\end{array}$$

(4)

The parameter settings of the image random masking improved algorithm in this paper are shown in

Table 1. Random masking parameters.

Parameter	Value
sl	0.04
sh	0.3
p	0.5
rc	0.3

. By improving the random masking method, the position of each masking can be covered as much as possible in the facial area, and the improved method is compared with the unimproved method, as shown in Figure 2.

3.2. Design a Clip_K5_ ShuffleNet Inverted Residual Structure

3.2.1. Inverted Residual Structure

The general standard convolution has two main functions: one is to tune-up the size of the upper layer’s feature map, and the other is to tune-up the number of the upper layer’s feature maps (adjust the number of channels). Depth-separable convolution makes some changes to the standard convolution, which is composed of depthwise (DW) convolution and pointwise (PW) convolution. Given the same input as standard convolution, it outputs the same result as the standard convolution after two steps of sequential operation. But the number of parameters and the calculation cost are relatively decreased. Theoretically, the computational cost of the standard convolution is 8–9 times that of the depth-separable convolution, which largely contributes to the light weight of the network. The inverted residual structure is used in Mobilenet_V2 [27], as shown in Figure 3, which first increases the dimensionality of the input feature map using 1 × 1 convolution, performs a convolution operation using 3 × 3 DW convolution, and finally uses 1 × 1 convolution to reduce its dimensionality, to reduce the amount of convolution operations while also reducing the loss of feature information.

The inverted residual structure is continued in ShuffleNet [28], and some networks’ parameters and calculations are reduced by stacking lightweight operators (DW convolution, etc.). And the inverted residual module in this network is divided into two structures: BasicBlock and DownBlock. There are 3 basic units (Stage2, Stage3, Stage4) in ShuffleNet, and each stage is repeatedly connected by a DownBlock and several BasicBlocks. As shown in Figure 4 (taking Stage2 as an example), the DownBlock needs to downsample and increase the dimension, so the input is copied into two copies, which are merged together after branch1 and branch2, respectively, and the channels are cross-rearranged to improve the information reuse of features. BasicBlock divides the input into two parts, and one part is directly merged and rearranged with the part of branch1 after the convolution extraction feature of branch2, to enhance the information communication between the two branches.

3.2.2. Improved ShuffleNet_Block

In the inverted residual structure of ShuffleNet, to compensate for the lack of the inter-channel information fusion function in DW convolution, a 1 × 1 convolution is used before and after the DW convolution on branch2. There is no need for dimensionality enhancement or reduction, and there is no obvious effect on improving model efficiency. Therefore, to pursue a more lightweight network, the 1 × 1 convolution in the dashed box in Figure 4 is cropped to reduce the network redundancy and is designed to implement the Clip_ShuffleNet in this paper. Compared with the 1 × 1 convolution, DW convolution has low computational complexity in networks, and Peng et al. [29] found that large convolution kernels have a stronger characterization ability for feature information. So, the convolution kernels of DW convolution in ShuffleNet are expanded in this paper, as shown in Figure 5, all 3 × 3 DW convolutions are expanded to 5 × 5 DW convolutions on the basis of Clip_ShuffleNet, and then the Clip_K5_ShuffleNet is designed and implemented.

Since the ReLU activation function will suppress the neurons when the input is negative, resulting in the model weights not being able to update, there is the issue of gradient disappearance, which affects the expression of networks. Therefore, the ReLU function is replaced by the β-Mish activation function in our paper. The β-Mish function uses α and β factors to normalize the region below the boundary of the Mish [30] function. β-Mish is a smooth, continuous, and non-monotonic activation function that uses the self-gating property to retain some negative information while eliminating the hard zero boundary of ReLU. And the function also plays a good role in the flow of the gradient and enhances the nonlinear expressive ability of networks. To some extent, the use of this function reduces the overfitting phenomenon and promotes the recognition accuracy and generalization ability of the network. The definition of the β-Mish function is as follows, where the value of parameter α depends on parameter β, α/β = 1/5, and the value of parameter β is between 1 and 200. In order to avoid saturation and slow down the training speed, this paper takes α = 7 and β = 35. The β-Mish function formula is expressed as follows:

$$F(x)=x\cdot \tanh (\ln (1+e^{\frac{\alpha x}{\sqrt{\beta +x^2}}}))$$

(5)

3.3. Shuffle Attention Module (SA)

The extraction of facial expression features is the most critical step in FER, which has a significant impact on recognition accuracy. The key to the extraction of facial expression features is the extraction of local detail features, such as the eyebrows, eyes, mouth, and other crucial parts that can best distinguish different expressions, and the human eye also pays more attention to these parts when discriminating expressions. The attention mechanism, as a theory proposed by a human-like cognitive behavior, focuses attention on more important information, and it can also quickly filter out more critical information from a lot of complex information and promote the efficiency of task processing. To further promote the accuracy of lightweight networks in FER, this paper introduces a lightweight and efficient Shuffle Attention module (SA) [31], which uses the replacement unit to efficiently fuse the channel and spatial attention mechanism, in exchange for a higher model recognition accuracy with a small increase in computational effort. Figure 6 shows the overall structure of the SA module, and the module mainly contains four points: Feature Grouping, Spatial Attention, Channel Attention, and Aggregation. Among them, Feature Grouping groups the input features, and each group of features is split into a Spatial Attention branch and Channel Attention branch along the channel dimension, which are used to learn channel features and spatial features, respectively. Aggregation fuses the two branches through ‘Concat’ and communicates feature information between groups through channel replacement operations to improve model performance.

3.4. The Dual-Branch Network (RLR-CNet)

With the advancement of study in CNNs, there has been remarkable progress in image classification, which is inseparable from lots of scientific, reasonable, and high-quality datasets, such as MS COCO, ImageNet, and other datasets. These high-quality datasets have a relatively uniform distribution of sample sizes for each category, which is conducive to improving the representation ability of the network for feature extraction. However, most classification datasets (such as facial expression dataset) in real scenes have an uneven distribution of sample sizes between classes, and exhibit a long-tail distribution [32] (a few classes have larger sample sizes and most classes have smaller sample sizes), to some extent, so the accuracy of model classification and recognition is affected. When extracting complex and subtle features such as facial expression features, it is especially critical to promote the quality of the dataset and the rationality of the sample distribution. Therefore, in order to resolve the issue of the uneven inter-class distribution of facial expression data and further promote the accuracy of FER in lightweight networks, RLR-CNet is proposed in this paper. This network has two branches based on fusion cumulative learning.

Specifically, the two branches proposed in this paper are called the “feature learning branch” and “class-rebalancing branch”. Figure 7 shows a schematic diagram of the dual-branch network. The Clip_K5_ShuffleNet inverted residual structure is used into two branches [33], the lightweight ShuffleNet as the backbone network is used in the dual-branch network, and all blocks in Stage2, Stage3, and Stage4 are replaced with the improved Clip_K5_ShuffleNet modules in this paper. All weights of the network before Stage4 are shared between the two branches. In each convolutional layer, the input images are convolved with the same filter to achieve weight sharing and cut down the number of parameters. Good feature learning is beneficial for rebalancing the learning of class-rebalancing branches and can greatly reduce the network’s computational load. For the two branches, the conventional uniform sampler and the reverse sampler are used, respectively, to obtain two data samples $(x_1,y_1)$ and $(x_2,y_2)$, which are used as the input of the feature learning branch and class-rebalancing branch, respectively, and the feature vectors $f_1$ and $f_2$ are obtained after the corresponding branch convolution and global average pooling. Considering that class-rebalancing can significantly boost classifier learning, it also impairs feature learning to some extent [34]. So, this paper introduces a specific method of cumulative learning, which staggers the learning “attention” of the two branches during network training. In other words, focusing on the learning of the feature learning branch in the early stage and the class-rebalancing branch in the later stage, to eliminate the effect of class-rebalancing on feature learning, achieves the goal of promoting the recognition accuracy of the network. Specifically, an adaptive trade-off parameter $\alpha$ is used to control the weights of $f_1$ and $f_2$, and the weighted feature vectors $\alpha \ast f_1$ and $(1-\alpha )\ast f_2$ are input into the classifiers $W_1$ and $W_2$, which are finally integrated together by channel merging. The output formula is as follows:

$$z=\alpha W_1^T{f}_1+(1-\alpha )W_2^T{f}_2$$

(6)

where $z$ is the final output and the predicted probability for each class $i\in \left\{1,2,\cdot \cdot \cdot ,C\right\}$ can be calculated by the following formula:

$${\widehat{p}}_i=\frac{e^{z_i}}{{\sum }_{j=1}^C{e}^{z_j}}$$

(7)

Notated as $\widehat{p}={\left|{\widehat{p}}_1,{\widehat{p}}_2,\cdot \cdot \cdot ,{\widehat{p}}_C\right|}^T$, the final category with the maximum probability is used as the last recognition result of the dual-branch network.

The dual-branch network designed in our paper mainly has the following three characteristics:

• The improved Clip_K5_ShuffleNet module is used in the whole of the dual-branch in this paper. Among them, the clipping of the 1 × 1 convolution reduces the model’s parameters, and the use of 5 × 5 DW convolution is beneficial for extracting global features. Pairing the β-Mish activation function in each block enhances the flow of feature information. Moreover, in order to obtain the expressions’ key features to ensure the validity of feature information, this paper incorporates the lightweight SA module in the feature learning branch but, in order not to add too many additional parameters as much as possible, only embeds the SA module between Stage 2 and Stage 3. These designs enable the model to promote the accuracy of recognition and classification while ensuring light weight.
• Weight sharing in the dual-branch network: The two branches share the weights of the network before Stage 4. On the one hand, the good learning of the feature learning branch is conducive to the learning of the class-rebalancing branch; on the other hand, the shared weights greatly reduce the computational complexity of the network and the speed of the model training, which in turn improves the recognition efficiency of the network.
• The dual-branch network that combines cumulative learning: Parameter $\alpha$ is set to control the weights and loss functions for the two branches, and realize the transfer of learning “attention” between two branches. In this way, the influence of class-rebalancing on feature learning is eliminated, the recognition accuracy of small sample size classes is enhanced, and the recognition accuracy of the network is comprehensively improved. Where parameter $\alpha$ is adaptively adjusted according to the number of iterations for training, indirectly determined by the total training time $T_s$ of the network and the current training time $T$, the formula is as follows:

  $$\alpha =1-{\left(\frac{T}{T_s}\right)}^2$$

  (8)

4. Experiment and Analysis

4.1. Experimental Preparation and Evaluation Indicators

4.1.1. Experimental Preparation

To verify the accuracy and effectiveness of the proposed lightweight FER model based on class-rebalancing fusion cumulative learning, this paper conducts ablation and comparative experiments on the FER2013, CK+, and RAF-DB datasets. Especially, in order to ensure the repeatability of the method proposed in this paper, the ablation, comparative, and other experiments are conducted multiple times under the same experimental equipment and parameter conditions, and the error rates of the final experimental results are all below 0.009%, which better proves the repeatability of the method proposed in this paper. And in order to further prove the reusability of the method in this paper, all the experimental results are verified under the conditions of the same experimental configuration and different experimental equipment, and the error rates of the experimental results are also kept below 0.01%, proving that the method in this paper has great reusability.

The experimental training and testing are based on the PyTorch deep learning framework on Pycharm. The configuration of the experimental server is as follows: Win10 operating system, Intel Core i5-12490F with 2.9 GHz CPU and 32 GB RAM, and NVIDIA GeForce RTX 3080 (10 GB) graphics card. Additionally, the settings of experimental parameters are shown in

Table 2. Experimental parameter settings.

Parameter	FER2013	CK+	RAF-DB
Loss function	Cross Entropy	Cross Entropy	Cross Entropy
Learning rate	0.01	0.01	0.01
Optimizer	SGD	SGD	SGD
Batch size	16	16	16
Momentum	0.9	0.9	0.9
Learning rate decay	0.5/50	0.5/50	0.5/50
Epochs	300	300	300

.

4.1.2. Evaluation Indicators

The evaluation indicators commonly used by machine learning for classification models are as follows: accuracy and confusion matrix (also known as the error matrix). The accuracy refers to the proportion of the correct number of samples output by the model to the total number of samples, which can be expressed as

$$Acc=\frac{TP+TN}{TP+FP+TN+FN}$$

(9)

where TP, TN, FP, and FN represent the number of samples of True Positive, True Negative, False Positive, and False Negative, respectively. Obviously, the sum of these four is the total number of samples. The four indicators of TP, TN, FP, and FN are presented together in a table called a confusion matrix, which can analyze the misclassification of each category.

The size of the model is generally measured by the number of parameters and corresponds to spatial concepts and spatial complexity. Because of the large number of parameters in many models, they are usually measured in units of megabytes (MB). This can be represented as

$$para{m}_{conv}=k_w\ast k_h\ast c_{in}\ast c_{out}$$

(10)

$$para{m}_{fc}=n_{in}\ast n_{out}$$

(11)

where $k_w\ast k_h$, $c_{in}$, $c_{out}$, $n_{in}$, and $n_{out}$ represent the size of the convolution kernel, the number of input channels, the number of output channels, the number of input channels of the dense layer, and the number of output channels of the dense layer, respectively. Finally, the parameter size of each layer is added together to obtain the total number of parameters.

4.1.3. Experimental Datasets

To verify the validity of the proposed model, we conduct experiments using FER2013, CK+, and RAF-DB datasets. These datasets contain different data scales and image complexity.

The FER2013 dataset contains a total of 35,886 facial expression samples. In this paper, the original sample is expanded to 60,000 based on the promoted random masking method. Among them, there are 46,000 training sets and 14,000 testing sets, the size of each sample is 48 × 48, and all samples are composed of grayscale images. And the dataset contains seven expressions of disgust, fear, anger, sad, happy, surprised, and neutral.

The CK+ dataset contains a total of 123 participants and 593 image sequences, all of which were collected under certain laboratory conditions. It is an upgraded version of the Cohn Kanda dataset with seven expressions. In this paper, the promoted random masking method is used to expand its samples to 1500, including 1050 training sets and 450 testing sets.

The RAF-DB dataset contains a total of 29,672 facial expression samples, all of which were sourced from real-life facial expression datasets. This dataset contains seven basic emoticon labels and twelve composite emoticon datasets. This paper selects a basic expression dataset for the experiment, and expands the basic expression sample to 23,008 pieces through the promoted random masking method, including 16,106 training sets and 6902 testing sets.

4.2. Ablation Experiments

To show the availability and rationality of our method, ablation experiments are performed for each module, and the experimental results are shown in

Table 3. Performance comparison of different modules of network.

Model	FER2013	CK+	RAF-DB	Parameter
ShuffleNet	65.71%	95.23%	84.31%	1.24 MB
ShuffleNet + RM	67.87%	95.91%	85.11%	1.24 MB
ShuffleNet + RM + Clip_K5	67.84%	95.87%	84.98%	0.94 MB
ShuffleNet + RM + Clip_K5 + SA	69.35%	96.35%	86.44%	0.96 MB
RLR-CNet (ours)	71.14%	98.04%	87.93%	1.02 MB

. RM denotes the promoted random masking method, Clip_K5_ShuffleNet denotes the improved inverted residual structure, SA denotes the shuffle attention mechanism module, RLR-CNet denotes the dual-branch network with cumulative learning, and RM + Clip_K5_ShuffleNet + SA + RLR-CNet denotes the lightweight network proposed in our paper.

Firstly, using ShuffleNet as the base network, the training samples are input into the network through a promoted random masking operation, and in order to prevent information loss and accelerate the speed of network operation, the lightweight ShuffleNet is improved to obtain a lighter Clip_K5_ShuffleNet network. To further extract local facial expression detail features, the SA module is integrated into the network to redistribute the feature weights of facial expressions from the two channel and spatial dimensions. To ensure the practicability and effectiveness of the network in real scenes, RLR-CNet is proposed in this paper. The effectiveness of each improvement module in the network is shown in Table 3.

In Table 3, it can be seen that after RM processing, the recognition accuracy of ShuffleNet on FER2013, CK+, and RAF-DB datasets increases by 2.16%, 0.68%, and 0.80%, respectively. The introduction of Clip_K5_ShuffleNet leads to a slight decrease in the recognition accuracy of the network, but on the other hand, it reduces the number of parameters, which is beneficial for accelerating the operation speed of the model. The introduction of the SA module further improves the recognition accuracy of the network.

Compared to the original network, our RLR-CNet promotes the recognition accuracy of the network by 5.43%, 2.81%, and 3.62%, and reduces the number of parameters by 17.74%. This shows that our method in the paper has some advantages in recognition accuracy and lightweight parameters.

The experimental training process of the proposed method in this paper is shown in Figure 8. For Figure 8a, the recognition accuracy growth of the model slows down at the 100th epoch, and the accuracy gradually becomes stable when it reaches the 200th epoch, with the highest accuracy reaching 71.14%. For Figure 8b, the recognition accuracy of the model rapidly increases at the beginning of the training, and when it reaches the 150th epoch, the accuracy tends to stabilize, with the highest accuracy reaching 98.04%. For Figure 8c, the recognition accuracy of the model continues to increase, and when it reaches the 180th epoch, the accuracy gradually becomes stable, with the highest accuracy reaching 87.93%. For Figure 8, the accuracy of the training set is always less than or equal to the accuracy of testing sets, and the accuracy of the two sets is generally very close, indicating that the method proposed in this paper can better capture the data characteristics and fit the data, and has better generalization.

For exploring more accurately the recognition performance of our method for FER, the confusion matrices are drawn based on the experimental results on the FER2013, CK+, and RAF-DB datasets, as shown in Figure 9. For the FER2013 dataset, the recognition accuracy of the happy and surprised expressions is relatively high, both exceeding 80%. For the CK+ dataset, all facial expressions’ recognition accuracies are above 95%. For the RAF-DB dataset, the happy, surprised, sad, and neutral expressions all have comparatively high recognition accuracy. Overall, on the three datasets, the recognition accuracy of happy and surprised expressions is relatively high, while the recognition accuracy of angry expressions is relatively low. This is because happy expressions often have significant features such as raised corners of the mouth and wrinkles around the eyes that are easy to recognize, and surprised expressions have recognized features such as an open mouth and wide eyes. Negative expressions such as anger and fear have strong similarities, which make it difficult to distinguish subtle changes, resulting in low recognition accuracy. Overall, the proposed method achieves good recognition performance for various facial expressions on the FER2013, CK+, and RAF-DB datasets.

4.3. Comparative Experiment of Mainstream Algorithms

To prove the effectiveness of the RLR-CNet proposed in this paper for FER, a comparative experiment is carried out with several mainstream algorithms such as ResNet18, ResNet50, VGG16, VGG19, and AlexNet from the aspects of the number of parameters of the model and the recognition accuracy. The experimental results are shown in

Table 4. Comparison experiments of mainstream algorithms.

Model	FER2013 (%)	CK+ (%)	RAF-DB (%)	Parameter (MB)
ResNet18	70.09	89.39	84.10	11.69
ResNet50	71.26	92.46	86.01	25.56
VGG16	68.89	95.46	81.68	14.75
VGG19	68.53	92.18	81.17	20.06
AlexNet	67.51	87.59	55.60	60.92
ours	71.14	98.04	87.93	1.02

.

For the FER2013 dataset of the facial expressions’ recognition accuracy, the proposed method in our paper achieves the highest accuracy among all the models, with an improvement of approximately 1.05% compared to ResNet18. For the CK+ dataset, the recognition accuracy of VGG16 is higher than those of other mainstream networks, while the proposed method in our paper achieves an improvement of 2.58% compared to VGG16. For the RAF-DB dataset, the recognition accuracy of the proposed method in our paper reaches 87.93%. The improvement in the recognition accuracy further verifies the effectiveness and strong generalization ability of the proposed method. In terms of model parameters, the parameter size of the proposed model is 1.02 MB, which is lowest among the mainstream algorithms. In general, while achieving a lightweight model, our method ensures great recognition accuracy, and then it demonstrates the efficiency and superiority of this paper’s method.

To further prove the effectiveness of our method, we conduct comparative experiments with some existing advanced methods on FER2013, CK+, and RAF-DB datasets. The advanced methods mainly include DCN and Inception V4, which are very novel in recent years, as well as lightweight networks such as Mini-Xception, MFN, and MANet, and classification networks embedded with modules such as SE and CBAM attention mechanisms. The experimental results are shown in

Table 5. Performance comparison of different methods on FER2013 dataset.

Model	Accuracy (%)
MANet [35]	69.46
Inception V4 [36]	66.80
DCN [37]	69.30
Minace [38]	70.20
ours	71.14

,

Table 6. Performance comparison of different methods on CK+ dataset.

Model	Accuracy (%)
DeRL [39]	97.30
APRNET50 [40]	94.95
ResMasking [41]	98.46
DTAGN [42]	97.25
ours	98.04

and

Table 7. Performance comparison of different methods on RAF-DB dataset.

Model	Accuracy (%)
Mini-Xception [43]	76.26
MFN [44]	85.39
SCN [45]	87.03
LA-Net [46]	87.00
ours	87.93

, respectively. It can be observed from the experimental results that, for the FER2013 dataset, the recognition accuracy is above 65%, while our method achieves an accuracy of 71.14%. For the CK+ dataset, the recognition accuracy is above 94%, while our method achieves an accuracy of 98.04%. For the RAF-DB dataset, the recognition accuracy is above 75%, while our method achieves an accuracy of 87.93%. This further proves that our method can ensure great accuracy in FER under the condition of lightweight implementation.

5. Conclusions

Aiming at the problems of a large number of parameters and a low accuracy of the current FER model, a lightweight model of FER based on class-rebalancing fusion cumulative learning is proposed in this paper. Through the proposed RLR-CNet, the problem where the model’s recognition accuracy is affected, which is due to the imbalanced inter-class distribution of facial expression data, is solved, and the reduction in the model’s parameters also speeds up the operation speed of the model to a certain degree. And the embedding of the SA module enhances the ability of the model to extract local details of facial expressions, and then improves the accuracy of FER on the lightweight network as a whole. According to the experimental results, the accuracy of the proposed method is 71.14% on the FER2013 dataset, 98.04% on the CK+ dataset, and 87.93% on the RAF-DB dataset. The model contains fewer parameters and achieves great recognition accuracy while implementing a lightweight network, and its accuracy is better than most current mainstream algorithms, demonstrating better effectiveness and applicability. In this paper, the object of study is the single-label expression dataset. The single-label expression can well represent the emotions contained in various expressions and has a good recognition accuracy for common expression categories, but in real scenes, there are still some minor expression categories that are not taken into account, such as tension and pride. The recognition of these minor expressions needs to be achieved by studying composite multi-label expressions. Therefore, in future research, more attention should be paid to the study of compound multi-label facial expression recognition.