LCA-GAN: Low-Complexity Attention-Generative Adversarial Network for Age Estimation with Mask-Occluded Facial Images

Abstract

Facial-image-based age estimation is being increasingly used in various fields. Examples include statistical marketing analysis based on age-specific product preferences, medical applications such as beauty products and telemedicine, and age-based suspect tracking in intelligent surveillance camera systems. Masks are increasingly worn for hygiene, personal privacy concerns, and fashion. In particular, the acquisition of mask-occluded facial images has become more frequent due to the COVID-19 pandemic. These images cause a loss of important features and information for age estimation, which reduces the accuracy of age estimation. Existing de-occlusion studies have investigated masquerade masks that do not completely occlude the eyes, nose, and mouth; however, no studies have investigated the de-occlusion of masks that completely occlude the nose and mouth and its use for age estimation, which is the goal of this study. Accordingly, this study proposes a novel low-complexity attention-generative adversarial network (LCA-GAN) for facial age estimation that combines an attention architecture and conditional generative adversarial network (conditional GAN) to de-occlude mask-occluded human facial images. The open databases MORPH and PAL were used to conduct experiments. According to the results, the mean absolution error (MAE) of age estimation with the de-occluded facial images reconstructed using the proposed LCA-GAN is 6.64 and 6.12 years, respectively. Thus, the proposed method yielded higher age estimation accuracy than when using occluded images or images reconstructed using the state-of-the-art method.

1. Introduction

In general, age, gender, expression, and race can be derived from facial appearances [1]. Age estimation is being increasingly used in diverse fields, such as statistical marketing analysis based on age-specific product preferences, medical fields such as the beauty industry and telemedicine, and age-based suspect tracking in intelligent surveillance camera systems [2]. Despite continuous efforts in research, including design of age estimation algorithms and models, data collection, system performance tests, and valid evaluation protocols, improving the accuracy of age estimation remains a challenge [3]. Age estimation is challenging because the human face is influenced by internal factors (size, wrinkles, shape, texture, race, etc.) and external factors (health, dietary habits, culture, environment, etc.), and these change over time through complex processes [4]. However, there are general and common features that can explain human facial aging [5]. Age estimation is divided into feature representation, extraction, and age learning stages. Though previous studies used various handcrafted feature-based methods, they require accurate prior knowledge of experts. However, there are no methods to verify the accuracy of the prior knowledge [6]. Unlike conventional methods, a convolutional neural network (CNN) can extract clear and robust facial features and learn the age on its own [7]. Generally, a CNN for age estimation consists of a convolutional layer and a multi-layer perceptron (MLP). The convolutional layer extracts and represents age information features from the facial images, and the MLP estimates the age with the represented features. Then, the distance between the estimated age and the age label is calculated by the loss function and then backpropagations are performed. The whole process is performed automatically, and unlike conventional methods that rely on prior knowledge, it can utilize information that cannot be extracted with prior knowledge. It provides better age estimation performance than traditional methods, and numerous researchers are actively conducting research on improving the results and accuracy.

Despite the advances in age estimation research, there are several problems when age estimation is applied practically. Facial images obtained in real unrestricted environments frequently have problems that degrade image quality related to resolution, illumination, noise, and occlusion. In particular, mask-occluded facial images have recently become frequent following the COVID-19 pandemic. Masks are worn more frequently because of hygiene, personal privacy concerns, and fashion. Mask-occluded facial images do not contain important features and information for age estimation, which reduces the accuracy of age estimation. Existing de-occlusion research has investigated masquerade masks that do not completely occlude the eyes, nose, and mouth [8,9,10,11], but no studies have investigated the de-occlusion of masks that completely occlude the nose and mouth, which is the goal of this study. To this end, this study proposes a novel low-complexity attention-generative adversarial network (LCA-GAN) for facial age estimation that combines an attention architecture and conditional generative adversarial network (conditional GAN) to de-occlude mask-occluded human facial images. Our innovation is for facial image de-occlusion. The present study is new compared to previous studies in four ways:

• This is the first study of its kind on age estimation that considers the de-occlusion of facial images where the nose and mouth are completely occluded by a mask;
• We propose a novel LCA-GAN for mask de-occlusion. LCA-GAN contains low-complexity attention blocks (LCABs) that reduce computation and complexity by combining down and upsampling with the attention module. LCAB comprises low-complexity channel attention (LCCA) and low-complexity spatial attention (LCSA), and it uses attention to assign weights based on the importance of features in channel and spatial dimensions;
• To reconstruct the facial feature information lost by mask occlusion as much as possible in de-occlusion, edge loss and content loss in LCA-GAN were used;
• The trained LCA-GAN and CNN for age estimation and experimental mask generated facial images were published [12], enabling a fair comparison with the performance of other researchers.

The structure of this paper is as follows. Section 2 analyzes existing age estimation studies that have used facial images and de-occlusion methods. Section 3 explains the overall experimental method and LCA-GAN, the de-occlusion network proposed in this paper. Section 4 presents a comparison of the performance and age estimation results using de-occluded images between existing de-occlusion methods and LCA-GAN based on the MORPH and PAL databases. As a final section, Section 5 concludes the paper.

2. Related Works

Facial images contain biological information with diverse attributes, such as race, gender, age, environment, and lifestyle. In [13], the distributions of these attributes and their averages were analyzed, and a method was presented to evaluate the bias of appropriate algorithms and databases. It has influenced various studies using human facial images [14,15,16]. However, it is difficult to investigate handcrafted feature-based age estimation, which requires consideration of diverse factors. Consequently, most age estimation studies have used CNNs since the emergence of deep learning. As shown in

Table 1. Classification and comparison of existing age estimation methods.

Categories	Method	Database	MAE	Exact	1-Off	ϵ-Error
Classification of multi-class ages	DEX [21]	IMDB-WIKI + LAP2015	3.22	N.A.	N.A.	0.26
	Residual DEX [22]	LAP2015	4.45			N.A.
	Dimensionality reduction + FFNNs [23]	WIKI + AmI-Face + Adience	3.30
	4C2FC [24]	MORPH	N.A.	46.39
	RoR [25]	IMDB-WIKI + Adience		67.3	97.51
	DEX [8]	MORPH	2.68	N.A.	N.A.
		FG-NET	3.09
		CACD	6.52
		IMDB-WIKI + LAP2015	N.A.	64.0
		Adience		N.A.	96.6	0.26
	4C2FC + dropout [9]	Adience	N.A.	84.8	89.7	N.A.
Regression based on metrics	3NNR [26]	Adience + MORPH + LAP2015	N.A.	N.A.	N.A.	0.37
	OR-CNN [27]	AFAD MORPH	3.34 3.27			N.A.
	VGG + BridgeNet [28]	MORPH FG-NET LAP2015	2.38 2.56 2.98
						0.26
Learning by the distribution of deep label	DLDL-v2 [29]	LAP2015 LAP2016 MORPH	3.14 3.45 1.97			0.272
						0.267
						N.A.
	Inception v4 [30]	MORPH FG-NET	1.32 2.19
Ranking	Ranking-CNN [31]	MORPH	2.96
	ODFL + OHRank [32]	MORPH FG-NET LAP2016 Adience	3.12 3.89 4.12 N.A.
						N.A.
						0.34
				54.0	88.2	N.A.
	ODL [33]	MORPH FG-NET LAP2016	2.92 3.71 3.95	N.A.	N.A.	N.A.
						0.312
Hybrid methods	Kernel ELM + CNN [34]	LAP2016	N.A.			0.37
	MRCNN [35]	MORPH	3.48			N.A.
	GA-DFL [36]	MORPH FG-NET LAP2015	3.25 3.93 4.21			N.A. N.A. 0.37
	CNN + ELM [37]	MORPH Adience	3.44 N.A.	N.A. 52.3		N.A.
	RAGN [10]	IMDB-WIKI + MORPH IMDB-WIKI + Adience IMDB-WIKI + LAP2016	2.61 N.A. N.A.	N.A. 66.5 N.A.		N.A.
						0.37
	AgeNet + divide and rule [11]	FG-NET MORPH IMDB-WIKI	4.02 3.48 3.29	N.A.		N.A.
	MA-ShuffleNet v2 [38]	MORPH FG-NET	2.68 3.81

, age estimation methods are generally classified into five categories [6]. Multi-class classification yields high age estimation performance when using limited resources and data with a single model. Hybrid methods, which combine multiple methods, supplement the shortcomings of combined models and yield high age estimation performance for large quantities of data in environments with many available computing resources. To measure the age estimation accuracy, researchers have used mean absolute error (MAE) [17], exact accuracy [18], 1-off [19], and normal score (ϵ-error) [20], among which MAE is the most commonly used.

As listed in Table 1, age estimation studies have used images obtained in restricted environments, such as the MORPH [39] and FG-NET [40] databases, and those from unrestricted environments such as IMDB-WIKI [41], Adience [42], LAP2015 [43], LAP2016 [44], and CACD [45]. MORPH is a database of human facial mugshot images with resolutions ranging from 640 × 480 to 1024 × 768 pixels. It contains various attributes, such as gender, age, and race, and is acquired in a restricted environment such as image resolution, illumination, and pose. FG-NET is a database of human facial images with gender and age information. The database collected facial images that satisfy specific conditions such as image resolution and pose from pictures of people with confirmed ages. In this case, the facial images that satisfy the conditions are similar to the facial images acquired in a restricted environment. On the other hand, the databases in unrestricted environment are collected from the internet, magazines, films, etc. These unrestricted databases contain facial images in natural poses, various image resolutions and illumination changes, and occlusions with various objects. Age estimation research using images from restricted environments has yielded relatively low age estimation accuracy for occluded images. Despite the difficulty of age estimation due to occlusion, previous age estimation studies have not considered the de-occlusion of facial images occluded by a mask that completely covers the nose and mouth. In addition, all datasets in Table 1 have a very small number of masked face images. Therefore, for our experiments, we generated a large number of masked face images from the MORPH and PAL databases.

Table 2. Comparison of the strengths and weaknesses of existing research and proposed methods in age estimation based on consideration of face occlusion.

Categories	Age Learning Technique	Method	Strength	Weakness
Age estimation without considering face occlusion	Handcrafted feature-based	Guo et al. [46]	Age estimation robust to restricted environment	They did not consider face-occluded images for age estimation
		Chen et al. [47]
	Deep feature-based	Inception v4 [30]
		MA-ShuffleNet v2 [38]
Age estimation with considering face occlusion		DEX [8]	Age estimation robust to occluded facial images	They trained simultaneously with occlusion and non-occlusion images, which made network convergence difficult
		AgeNet + divide and rule [11]
		RAGN [10]
		4C2FC + dropout [9]
		Proposed method		Additional procedures are required to train LCA-GAN

presents a comparison of studies that do and do not consider face occlusion with the proposed method.

A study [30] proposed recurrent age estimation (RAE), which combines inception-v4 [48] and long short-term memory networks (LSTM) [49]. It extracts features from facial images using inception-v4 and learns individual aging patterns with LSTM. To solve the problem of training overfitting, researchers have proposed label distribution learning (LDL), which uses the ambiguity between the label age and predict age. A study [38] proposed mixed attention-ShuffleNet-v2 (MA-SFV2), which combines mixed attention and ShuffleNet-v2 [50]. Classification, regression, and distribution methods were simultaneously applied to learning to transform the output layer of the base model. In [10], CNN2ELM, an ensemble model combining CNN and extreme learning machine (ELM), was proposed for leaning age. It achieved decent results in ChaLearn 2016 [51], a human age estimation competition. In [8], a deep expectation of apparent age (DEX) system was proposed, which uses the softmax expected value refinement of the VGG-16-based network [21]. More specifically, they defined the regression problem of age learning as a classification problem and performed age estimation by multiplying the age label and the class probability distribution, which is the last softmax output of VGG-16. A study [11] proposed the divide-and-rule architecture and AgeNet, a model based on the method of GoogleNet [52]. AgeNet is a feature extractor and divide-and-rule approach to age learning as an ordinal regression problem. As such, research on age estimation includes studies using restricted environment images [30,39,47,48], studies using unrestricted environment images [9,11], and studies using images of both environments [9,21,38]. This age estimation research includes many studies that excluded occlusion in restricted environments or ignored occlusion in unrestricted environments, so de-occlusion was not considered.

However, face occlusion frequently occurs in the real world and is challenging to solve through camera hardware. Most existing de-occlusion methods reconstruct synthesized images because of the lack of databases with non-occluded and occluded image pairs [53,54,55,56]. In [53], a two-stage occlusion-aware GAN was proposed, trained with 44 images occluded by sunglasses, hats, scarves, and phones with a random shape, location, and size in a face database, but it did not use facial mask images. This method removes occluded areas with the existing Pix2pix-based [57] GAN architecture and de-occludes the image only using information from non-occluded areas. However, this is an unrealistic occlusion condition, and while de-occlusion is successful for most objects, it fails for glasses and sunglasses. Previous research proposed a method to solve the face recognition problem associated with block occlusion of facial images by two robust feature-based representations, which were designed to fit the errors to a distribution described by a tailored loss function and the reduced rank structure of the errors relative to the image size [58]. The work in [59] proposed MRGAN, a GAN-based two-level network that removed the areas occluded by medical masks and reconstructed the removed areas. Stage one detected masks, and stage two performed de-occlusion. This method is based on a complex network and is computationally intensive. This experiment achieved good de-occlusion results but did not reconstruct color and detail information well in the occluded area. The work in [60] proposed a two-stage GAN-based method for de-occlusion of small objects in facial image such as microphones. Steps one and two were trained similarly to a conventional GAN, but the de-occluded image from step one was used as an input to step two to generate a robust de-occluded image. The de-occlusion results showed that the texture and detail were de-occluded well, but the outline of the occluded object remained.

Moreover, previous study [54] presented a two-stage method; in stage one, the occluded area is detected with an encoder–decoder structure and converted to a mask image as a pre-processing method, and in stage two, the occluded area is transformed through two conditional GAN [61] architectures. For supervised learning, face images were collected from the CelebA database, and the collected images were synthesized with the collected mask images occluding the eyes using Photoshop CC 2080 [54]. This method requires additional binary mask images to be trained, and the de-occlusion results fail on complex and detailed mask images. Another study [55] proposed Swap-R&R that compensates for the lack of paired databases. This method shows very robust de-occlusion performance with facial images occluded by glasses, sunglasses, makeup, and headsets, but it requires an additional 3D face reconstruction network in training and testing, and the network computation is very large [55]. In [56], they reconstructed identity-preserved and de-occluded facial images by CNN, which was supervised with identity labels. By using the additional channel for occlusion detection, a mask for occlusion is computed as a pre-processing method and combined with the reconstructed face. This experiment collected frontal face images between −45 and 45 degrees from the CASIA-WebFace database. The collected face images are used to synthesize multiple objects for supervised learning. There are more than 100 templates for occluding objects such as masks, glasses, and hands [56], and among them, objects such as glasses and masks require accurate position information. Objects that require such location information are synthesized directly using an image program, while other objects are synthesized randomly. However, they only deal with grayscale facial images and produce results including artifacts. Though de-occlusion research has been conducted on masquerade masks that do not completely occlude the eyes, nose, and mouth [54], no studies have investigated de-occlusion-based age estimation of masks that completely occlude the nose and mouth, which is the goal of this study. To solve the aforementioned problems, this study proposes LCA-GAN-based facial mask de-occlusion and an age estimation method using this technique.

3. Proposed Methods

3.1. Overview of Suggested Method

Figure 1 illustrates the entire procedure of robust age estimation for mask-occluded facial images proposed in this study.

Stages one and two entail pre-processing. First, the positions of the face and eyes are detected, and based on the detected positions, we execute the compensation of in-plane rotation, and the face region of interest (ROI) is re-defined. Following pre-processing, the mask-occluded image is de-occluded using the proposed LCA-GAN. The CNN-based age estimation method is then adopted to predict the age of the person in the de-occluded image.

3.2. Pre-Processing

As shown in Figure 2, the input images are processed. For pre-processing, this study used the dlib facial feature tracker [62], which extracted features with histogram of oriented gradients (HOG) and trained a linear classifier on the extracted information. It does not use any parameters or thresholds except for the upsample_num_time option, which is used for iterating the detection while scaling the input image, and we used the default value of one in our experiment. It was used on original face images of different sizes to detect face landmarks and the face box region. First, the dlib facial feature tracker for facial feature points [62] is used to locate the positions of the eyes, as shown in Figure 2b. Using Equation (1) based on the positions of both detected eyes, in-plane rotation compensation is performed, as shown in Figure 2c. In-plane rotation compensation is performed based on the center position between both eyes (green dot between the eyebrows in Figure 2b).

$$\theta ={tan}^{-1}\left(\frac{R_y-L_y}{R_x-L_x}\right)$$

(1)

($R_x$, $R_y$) and ($L_x$, $L_y$) represent the x- and y-axis positions of the centers of the detected right and left eyes, respectively. In the in-plane rotation compensated image, the face region is found using the dlib facial feature tracker, as shown in Figure 2c, and this face region becomes the face ROI, as shown in Figure 2d. To use the face ROI as an input for LCA-GAN, it is resized to 256 × 256 × 3 by bilinear interpolation.

3.3. De-Occlusion of Masked Facial Image by LCA-GAN

General image quality problems that distort image information include low resolution, blur, low illumination, noise, etc. However, regarding the occlusion problem, the occluding object reduces image information. Hence, unlike image quality problems where information can be directly extracted from the distorted area, for the occlusion problem, it is more difficult to directly extract information from the occluded area. Consequently, it is difficult to learn the mapping from occluded to de-occluded images using a CNN. To solve this problem, this study proposes LCA-GAN based on adversarial learning. Figure 3 shows the architecture of LCA-GAN. The generator is based on U-net [63]; by using LCAB, which combines up and downsampling with channel and spatial attention, it reduces complexity and computation. For the discriminator, a patch discriminator is used to output with a size of 30 × 30 × 1. It obtains probabilities for each of the 30 × 30 patches and determines whether each of the 900 local patch areas is a real or fake image. Finally, it averages all 900 probabilities to finally determine if the entire global image is real or fake.

3.3.1. Generator

Figure 3a shows the generator for de-occlusion in this study, which uses the U-net [63] architecture comprising an encoder–decoder and skip connection. In the encoder–decoder used for U-net’s continuous down and upsampling, the encoder extracts features, and the decoder learns the mapping for image patches corresponding to the extracted features. Additionally, it concatenates high-stage encoder block features with low-stage decoder block features to compensate for lost high-level information and to balance high- and low-level information. However, this architecture is inefficient for the occlusion problem, where it is difficult to directly extract information from the occluded area. To solve this problem, this study proposes LCAB, which combines an attention mechanism [64] with down and upsampling. LCAB is composed of LCCA and LCSA; attention is used to assign weights according to the importance of features in the channel and spatial dimensions. The next subsection describes LCAB architecture in detail. When de-occluding the occluded area, the generator uses edge loss to create a detailed and sharp image as well as content loss to maintain the information of the non-occluded area and the texture in the target image. In this de-occlusion process, ${\mathcal{L}}_1$ loss function was used for identity loss to maintain the information of the original image.

Table 3. Generator architecture in LCA-GAN.

Layer			Size of Feature	Concatenation
Input image			256 × 256 × 3	-
Encoder	Convolution layer 1		256 × 256 × 64	-
	Spatial attention		256 × 256 × 64	-
	LCAB 1	LCCA LCSA	128 × 128 × 64 128 × 128 × 128	-
	LCAB 2	LCCA LCSA	64 × 64 × 128 64 × 64 × 256	-
	LCAB 3	LCCA LCSA	32 × 32 × 256 32 × 32 × 512	-
	LCAB 4	LCCA LCSA	16 × 16 × 512 16 × 16 × 512	-
	LCAB 5	LCCA LCSA	8 × 8 × 512 8 × 8 × 512	-
Decoder	LCAB 6	LCCA Concatenation LCSA	16 × 16 × 512 16 × 16 × 1024 16 × 16 × 512	LCAB4
	LCAB 7	LCCA Concatenation LCSA	32 × 32 × 512 32 × 32 × 1024 32 × 32 × 512	LCAB3
	LCAB 8	LCCA Concatenation LCSA	64 × 64 × 512 64 × 64 × 768 64 × 64 × 256	LCAB2
	LCAB 9	LCCA Concatenation LCSA	128 × 128 × 256 128 × 128 × 384 128 × 128 × 128	LCAB1
	LCAB 10	LCCA LCSA	256 × 256 × 128 256 × 256 × 64	-
	Convolution layer 2 Tanh activation layer		256 × 256 × 3	-
Generated image			256 × 256 × 3

presents the overall architecture of the generator. As shown in Table 3, an image of 256 × 256 × 3 including the mask-occluded area is inputted to the generator of LCA-GAN, and a feature map of 256 × 256 × 64 is obtained via Convolution layer 1 and Spatial Attention. Then, by passing through LCAB 1~5 (including LCCA and LCSA), the feature map of 8 × 8 × 512 is obtained as the final output of encoder. This feature map is again passing through the decoder of LCAB 6~10 (including LCCA, Concatenation, and LCSA except for LCAB 10 including only LCCA and LCSA) and the upsampled feature map of 256 × 256 × 64 is obtained. Then, this feature map is passing through Convolution layer 2 and Tanh activation layer, and the final generated (mask-de-occluded) image of 256 × 256 × 3 is obtained as the output of decoder.

3.3.2. The Structure of LCAB

The attention mechanism abstracts the importance between the modalities of the represented features to incorporate high-level information. Representative attention methods used in images include spatial attention and channel attention [64]. Channel and spatial attention determine which features are important in the channel and spatial dimensions, respectively, and assign corresponding weights. In this study, channel and spatial attention were used to detect and de-occlude mask-occluded areas. Moreover, we proposed LCAB, which combines down and upsampling with the attention process to reduce the complexity and computation that increases due to attention and continuous processes after down or upsampling. LCAB comprises LCCA and LCSA, the structures of which are illustrated in Figure 4 and Figure 5.

Channel attention has two convolutional layers, average pooling, three multi-layered perceptrons (MLPs), a sigmoid layer, multiplication with convolutional layer features, and addition with input features passed through skip-connection. The importance of channel dimension features is arranged using the two-stage convolution layer to re-represent the input-represented features. The features are then compressed by global average pooling in the spatial space, and MLP is used to calculate the importance of modalities between the features in the channel space. Here, the first and third layers of MLP are equal to the channel dimension size of the input features, and the second layer is 1/4 the channel dimension size of the input features. Using sigmoid activation, attention is constructed from the importance of these arranged features. Following the convolutional layer, it is multiplied with the features and then added to the input features with high-level information. This study proposes LCCA, which combines down and upsampling with channel attention. In the two-stage convolutional layer, the first stage uses a 4 × 4 size filter to scale the spatial space by 2 in upsampling and by 1/2 in downsampling. The second stage uses a 3 × 3 size filter to maintain the size of the spatial space. Moreover, the deformation of high-level information is minimized using bilinear interpolation, and the skip-connection of the input features is passed.

As shown in Figure 5, spatial attention is composed of two convolutional layers, global average pooling, a convolutional layer, a sigmoid layer, multiplication with convolutional layer features, and addition with input features passed through skip-connection. The importance of spatial dimension features is arranged using the two-stage convolution layer to re-represent the input-represented features. The features are then compressed by global average pooling in the channel space, and the importance of modalities between the features in the spatial space is calculated through the convolutional layer. Using sigmoid activation, attention is created from the importance of these arranged features. Following the convolutional layer, it is multiplied with the features and then added to the input features with high-level information. This study proposes LCSA, which combines down and upsampling with spatial attention. The two-stage convolutional layer maintains the spatial space using a 3 × 3 size filter but adjusts the channel dimension to the desired size. After global average pooling, the convolutional layers use a 7 × 7 size filter to calculate the importance of modalities between features in the spatial space and then represent them as probability values using the sigmoid activation layer. Additionally, the size of the channel space of the input features is adjusted using bilinear interpolation, the deformation of high-level information is minimized, and the information is passed. Through LCCA and LCAS, channel and spatial attention preserve the original goal of representing the importance of the represented features from the perspectives of “what” and “where”, combining the down and upsampling processes. Moreover, they reduce computation and complexity caused by the use of continuous down and upsampling and attention mechanisms, and they are effective for the de-occlusion process.

3.3.3. Discriminator

LCA-GAN uses a patch discriminator. It receives a fixed input image and a random target image and output image, and then it concatenates them. Convolution is then performed, and each grid of extracted features has a receptive field according to the computational structure. In this study, individual grids of 1 × 1 × 1 units of the final output, which has a size of 30 × 30 × 1, have a receptive field of 70 × 70, use individual grids to determine the local area, and then distinguish the global area with the average of all grids. In this process, identity loss and content loss are used to maintain the continuity of information in the input image.

Table 4. Discriminator architecture in LCA-GAN. CL and BN mean convolution layer and batch normalization, respectively.

Layer		Size of Feature
Input image		256 × 256 × 3
Target or de-occluded image		256 × 256 × 3
Concatenate		256 × 256 × 6
CL 1	Convolution BN ReLU	128 × 128 × 64
CL 2	Convolution BN ReLU	64 × 64 × 128
CL 3	Convolution BN ReLU	32 × 32 × 256
CL 4	Zero padding Convolution BN Leaky ReLU	34 × 34 × 256 31 × 31 × 512
CL 5	Zero padding Convolution Sigmoid Average pooling	33 × 33 × 512 30 × 30 × 1 1 × 1 × 1
Output		Real or fake

lists the detailed structure of the discriminator.

LCA-GAN proposed in this study performs learning using pairs of images comprising the mask-occluded image (input image) and original un-occluded image (target image). Conditional GAN [57] learns the mapping using the loss function in Equation (2), which receives an input image $I^{In}$ and generates an output image $I^{Out}$ that is similar to the target image $I^{Target}$.

$${\mathcal{L}}_{GAN}\left(G, D\right)={\mathbb{E}}_{I^{In},I^{Target}}\left[logD\left(I^{In},I^{Target}\right)\right]+{\mathbb{E}}_{I^{In}}\left[log(1-D\left(I^{In},G\left(I^{In}\right)\right))\right]$$

(2)

This adversarial learning method generates smooth pixel-wise images [65]. In this study, the generator used content loss to maintain the texture of the original face when de-occluding the mask-occluded area. For this purpose, the ${\mathcal{L}}_2$ loss function of Equation (3) is applied to the VGG-16 (pre-trained with ImageNet) to measure the dissimilarity of $I^{Out}$ and $I^{Target}$.

$${\mathcal{L}}_{cont}={\mathbb{E}}_{I^{Out},I^{Target}}\left[{\left(I^{Target}-I^{Out}\right)}^2\right]$$

(3)

The proposed LCA-GAN preserves the area other than the mask-occluded area in the mask-occluded image and performs de-occlusion. The discriminator concatenates pairs of $I^{In}$ and $I^{Out}$ or $I^{In}$ and $I^{Target}$, which enables learning to reinforce high-frequency information in the mask-occluded area of $I^{Target}$. This is accomplished by applying the edge loss function in Equation (4) to generate a detailed de-occluded image. Δ denotes the Laplasian operation, and ε is 10⁻³.

$${\mathcal{L}}_{Edge}={\mathbb{E}}_{I^{Out},I^{Target}}\left[\sqrt{{\left(\Delta I^{Target}-\Delta I^{Out}\right)}^2+{\epsilon }^2}\right]$$

(4)

Rather than learning the data distribution of $I^{Target}$, adversarial learning sometimes strongly tends toward receiving the discriminator’s determination of the real image. To prevent this and preserve the identity of the image, we added identity loss, which uses the ${\mathcal{L}}_1$ loss function, as shown in Equation (5).

$${\mathcal{L}}_{Iden}={\mathbb{E}}_{I^{Out},I^{Target}}\left[\left|\left|I^{Target}-I^{Out}\right|\right|\right]$$

(5)

Finally, our final loss function is in Equation (6). Using the training data, 1.5, 2, and 2 were determined as the optimal values of ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$, respectively, to obtain the best age estimation accuracy.

$${\mathcal{L}}_{LCA}=\arg \underset{G}{min}\underset{D}{max}{\mathcal{L}}_{GAN}\left(G,D\right)+{\lambda }_1{\mathcal{L}}_{cont}+{\lambda }_2{\mathcal{L}}_{Edge}+{\lambda }_3{\mathcal{L}}_{Iden}$$

(6)

3.4. Age Estimator

The DEX model [8] was used to predict the age of de-occluded facial images with LCA-GAN, which exhibited good results in the Looking at People (LAP) 2015 [66] competition and previous research results on age estimation accuracy [67]. DEX is an age estimation model based on VGG-16 [68], an existing classification network. For age estimation, VGG16 pre-trained on ImageNet was additionally pre-trained using the IMDB and WIKI databases. Since human aging typically involves sequential changes over time, the similarity between adjacent classes in DEX is high, so the probability of the trained model is considered to have a normal distribution. Therefore, rather than estimating the class label showing the highest probability score as age, the age was predicted as the product of the class label and probability value, as shown in Equation (7).

$$Estimated age\left(I\right)={{\displaystyle \sum }}_1^n{l}_i{p}_i$$

(7)

where $I$ denotes the input facial image, $n$ indicates the number of classes, $l_i$ denotes the ith class label, and $p_i$ corresponds to the ith output probability value. The detailed age estimation methods of DEX are illustrated in Figure 6.

DEX [8] is a VGG-16-based network with a convolution layer, which consists of a convolution filter, batch normalization, an ReLU activation function, two MLP layers with 4096 nodes, and an output MLP layer equal to the size of the age class labels. The convolutional layers extract features with age information and represent the features for age learning. MLP layers learn age from the represented features. The last MLP layer outputs a probability value through a softmax activation function. Finally, to improve the age estimation performance, DEX applies the age estimation method described in Equation (7). In this experiment, we used the ${\mathcal{L}}_2$ loss function in DEX to emphasize the relation of close age classes, thus making the age estimation method using Equation (7) more robust.

4. Experimental Results

4.1. Data and Environment for Experiments

As shown in Figure 7, this study used MORPH [39] and PAL [69] as the databases to de-occlude mask-occluded facial images. Given the lack of open databases of mask-occluded facial images obtained in real environments that include existing age information, as shown in Figure 8, we generated mask-occluded facial images using mask images without a background image directly acquired from MORPH and PAL, which are existing human facial databases. Figure 8a shows the original facial image, and Figure 8b shows the mask image with no background. Subsequently, in the facial image, the dlib facial feature tracker [62] explained in Section 3.2 detects the eye area, as shown in Figure 8c. Using the center position of the eyes, the method described in Section 3.2 is used to perform in-plane rotation, as shown in Figure 8d; in this image, the dlib tracker for facial feature points finds the position of the face landmark corresponding to the annotated point of the mask image. Based on this position information, the annotated mask image is geometrically transformed and warped, as shown in Figure 8e. As shown in Figure 8f, the transformed mask image is then occluded on the aligned facial image, and the face ROI is detected again using the dlib tracker. The ROI is re-defined with the detected face ROI, and the final face ROI image of the mask-occluded image is created, as shown in Figure 8g.

The experiments were performed with two-fold cross validation, and in each fold, we used 5% of the training images as a validation set. To locate the face ROI, we used the python library (version 3.5.2) [70] and OpenCV (version 4.2.0) [71]. The specification of the desktop computer for our experiments is as follows: 3.5 GHz CPU (Intel^® Core™ i7-3770K) and 24 GB RAM, Windows Tensorflow (version 2.2.0) [72], and Nvidia graphics processing unit (GPU) card (Nvidia GeForce GTX 1070 [73]).

4.2. Training of LCA-GAN for Masked Image De-Occlusion and CNN for Age Estimation

LCA-GAN proposed in this study performs learning using the mask-occluded facial image as the input image and the original facial image without a mask as the target image. During training, through online augmentation, the input images were resized to 286 × 286 × 3 and then randomly cropped to 256 × 256 × 3. The adaptive moment estimation (Adam) optimizer [74] was used during training, with a learning rate of 0.0002, beta_1 of 0.5, and beta_2 of 0.999. Training was conducted for 100 epochs; Figure 9 shows the training and validation loss graphs of the generator and discriminator of LCA-GAN. It is evident that the generator and discriminator converged, indicating that the training data were sufficiently learned. For validation loss, the results of the generator and discriminator converged, indicating that LCA-GAN was not overfitted to the training data. In the case of GAN, mode collapse usually occurs when a generator tries to map an input (training data set) to the same output (generator function). The discriminator and generator should be learning together and interacting with each other, but one becomes too well trained (learning imbalance), and mode collapse occurs [75]. As shown in Figure 9, the discriminator becomes too well trained compared to the generator in our experiment, which is usually the case for conventional GAN [75,76,77]. Therefore, although the mouth areas are different in the input images, those in the generated output images are somewhat similar, which represents a small level of mode collapse. However, other areas of face in the generated output images were different according to our LCA-GAN, which confirms that overfitting and mode collapse were not severe in our LCA-GAN.

The slow convergence of the validation loss of the generator in Figure 9 was due to the insufficient number of MORPH databases used. In addition, considering the age labels ranging from 16 to 77 years old, various ethnicities, and unbalanced gender ratio in the MORPH database, the distribution of the data is very complex and unbalanced. These factors create a condition where the loss of validation data using only 5% of the training data can lead to late convergence. In this case, the difficulty of learning generally increases, and overfitting, underfitting, and divergence of losses are likely to occur. We described our training and validation sets according to the distribution of our experimental database in

Table 5. Distribution of our experimental database for two-fold cross validation.

	Gender	Age	White	Black	Hispanic	Asian	Other	Total
Training	Male	16~25	952	5834	401	44	4	7235
		26~35	864	4184	231	13	5	5296
		36~45	1090	4300	92	3	4	5490
		46~55	579	1973	24	4	7	2588
		56~65	90	268	2	0	0	360
		66~77	7	16	0	0	0	23
		Total	3582	16,574	750	63	20	20,990
	Female	16~25	283	710	20	5	0	1017
		26~35	367	761	19	0	2	1149
		36~45	395	818	6	0	5	1224
		46~55	106	275	0	0	1	383
		56~65	18	25	0	0	1	45
		66~77	1	1	0	0	0	2
		Total	1169	2591	46	6	9	3820
Validation	Male	16~25	212	1296	89	10	1	1608
		26~35	192	930	51	3	1	1177
		36~45	242	956	20	1	1	1220
		46~55	129	439	5	1	2	575
		56~65	20	60	0	0	0	80
		66~77	2	4	0	0	0	5
		Total	796	3683	167	14	4	4665
	Female	16~25	63	158	4	1	0	226
		26~35	82	169	4	0	1	255
		36~45	88	182	1	0	1	272
		46~55	24	61	0	0	0	85
		56~65	4	6	0	0	0	10
		66~77	0	0	0	0	0	1
		Total	260	576	10	1	2	849
Total	Male		7961	36,832	1667	141	44	46,645
	Female		2598	5757	102	13	19	8489

of Section 4.2. As shown in Table 5, there exists a class imbalance in some age ranges, races, and gender. To solve this problem, we increased the training and validation data by data augmentation that included translation, cropping, and horizontal flipping for classes whose numbers of images were much smaller than those of other classes (e.g., the images of ages from 56~65 and 66~77). From that, we could make all the images of each class distributed equally by gender, age, and race for the training and validation. In our experiment, the convergence of the validation loss was slightly delayed, as shown in Figure 9, but the result was well learned without overfitting the training data.

The reason why the validation loss of the discriminator increased after 10 epochs is as follows. The Adam optimizer used in this experiment has various advantages, but it has the disadvantages of poor conditioning problems and slow initial learning speed that depends on the size of the database, the value of the hyperparameter, and the loss function used [74]. In Figure 9, the slightly slower convergence speed of the generator loss is likely due to the initially slow learning speed of the Adam optimizer, while the relatively fast convergence of the discriminator loss is due to the fact that it is relatively easier to learn than the generator [76]. However, the brief increase in discriminator loss is a result of the poor conditioning problem mentioned above. The first and second moments used by the Adam optimizer are the mean of the sample mean and sample square of the input data, respectively. In this experiment, mean squared error (MSE) is used as the content loss function, which causes a poor conditioning problem in the second moment during back-propagation. Several studies have proposed methods to solve this problem, and in our paper, we applied L_2 regularization [78], used the largest possible batch size in the experimental environment [79], used a learning rate of 0.0002, which is smaller than the 0.001 usually used for the Adam optimizer [80], and applied a weight decay every 10 epochs [78]. Therefore, the discriminator loss in Figure 9 increases slightly after 10 epochs, decreases again after 50 epochs, and gradually converges, which can be seen as a good response to the problem of the Adam optimizer in our experiment.

Subsequently, the images de-occluded using LCA-GAN were learned by DEX [8], an age estimation CNN model. The same random cropping outlined above was applied through online augmentation, and learning was conducted for 200 epochs using the Adam optimizer, with a learning rate of 0.0002, beta_1 of 0.5, and beta_2 of 0.999 [74]. Figure 10 illustrates the training loss and accuracy graphs of DEX as well as the validation loss and accuracy graphs of DEX. The convergence of the training loss and accuracy graphs demonstrates that the DEX age estimator was sufficiently trained on the de-occluded training data generated by LCA-GAN. Moreover, the convergence of the validation loss and accuracy graphs demonstrates that the DEX age estimator was not overfitted to the de-occluded training data generated by LCA-GAN.

4.3. Testing with MORPH Database

4.3.1. Comparisons of the Quality of Images Generated by Proposed Method and State-of-the-Art Methods

To compare the performance of the de-occlusion model for mask-occluded facial images in this experiment with other models, the structural similarity index measure (SSIM) [81] and peak signal-to-noise ratio (PSNR) [82] were used to measure the similarity between the original image and the generated de-occluded image. SSIM is expressed in Equation (9), and PSNR is expressed in Equation (10). Larger values of both SSIM and PSNR indicate better performance of the de-occlusion model.

$$\mathrm{MSE}=\frac{1}{WH}{\displaystyle \sum }_{i=0}^{W-1}{\displaystyle \sum }_{j=0}^{H-1}{\left[I_o\left(i, j\right)-I_d\left(i, j\right)\right]}^2$$

(8)

$$\mathrm{SSIM}=\frac{\left(2{\mu }_d{\mu }_o+C1\right)\left(2{\sigma }_{do}+C2\right)}{\left({{\mu }_d}^2+{{\mu }_o}^2+C1\right)\left({{\sigma }_d}^2+{{\sigma }_o}^2+C2\right)}$$

(9)

$$\mathrm{PSNR}=10{log}_{10}\left(\frac{{255}^2}{\mathrm{MSE}}\right)$$

(10)

$I_o$ represents the original image, and $I_d$ represents the mask de-occluded image. Moreover, W and H denote the width and height of the image, respectively. ${\mu }_o$ and ${\sigma }_o$ indicate the mean and standard deviation of the pixel values of the original image, respectively. ${\mu }_d$ and ${\sigma }_d$ indicate the mean and standard deviation of the pixel values of a mask de-occluded image, respectively, and ${\mu }_{do}$ denotes the two images’ covariance. $C1$ and $C2$ correspond to the positive constant offsets.

As presented in

Table 6. Comparative SSIM and PSNR of original image and de-occluded images.

	LCA-GAN	AFD-StackGAN [54]	CFR-GAN [55]	MPRNet [83]	CycleGAN [84]	Pix2pix [57]
SSIM	0.6962	0.6769	0.7107	0.7031	0.5630	0.7225
PSNR (unit: dB)	19.0302	16.3121	18.3067	19.6427	19.3321	18.8731

, Pix2pix [57] and MPRNet [83] yielded the best performance for SSIM and PSNR according to the de-occlusion results, while the proposed LCA-GAN exhibit the fourth and third highest performance for SSIM and PSNR, respectively. Nevertheless, SSIM and PSNR are values that represent the image quality according to de-occlusion; the primary goal of this study is to improve age estimation accuracy (not image quality) through de-occlusion. As presented in Section 4.3.2 and Section 4.4.2, the proposed LCA-GAN yielded the highest age estimation accuracy.

4.3.2. Comparisons of Age Estimation Accuracy

Ablation Studies

As shown in Equation (11), we used the mean absolute error (MAE), the most frequently applied metric [85,86], to evaluate age estimation accuracy. A lower MAE value shows better performance of age estimation.

$$\mathrm{MAE}=\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n\left|p_i-y_i\right|$$

(11)

In the above equation, n denotes the number of images, $p_i$ denotes the predicted age, and $y_i$ denotes the ground-truth age. For the first ablation study, we compared the age estimation accuracy of the de-occluded images depending on the application of LCCA and LCSA, which form LCAB, and edge and content losses in the proposed LCA-GAN. According to the results in

Table 7. Ablation study with or without LCSA, LCCA, and Edge and Content losses in our LCA-GAN (unit: years).

LCSA	LCCA	Edge Loss + Content Loss	MAE
×	×	×	7.72
◯	×	×	7.11
×	◯	×	7.09
×	×	◯	7.83
◯	◯	×	6.82
◯	◯	◯	6.64

, the proposed LCA-GAN yields the highest age estimation performance in the generated de-occluded images when using LCCA, LCSA, and edge and content losses, respectively.

We performed an experiment to conduct an additional ablation study, as shown in

Table 8. Ablation study according to various backbone generators in LCA-GAN. Baselines 1 and 2 show the MAEs with original non-occluded and mask-occluded face images by DEX, respectively. Baseline 2 shows the MAE with original non-occluded face images by DEX. (Pix2pix* indicates the method that subtracts the original image and masked image, concatenates the input image with the image where only the occluded area remains, and uses this input image in Pix2pix learning) (unit: years).

Method	MAE
Baseline 1	5.80
Baseline 2	10.45
U-net	7.70
Pix2pix (LCA-GAN)	6.64
CycleGAN	7.15
Pix2pix*	6.91

. We compare the age estimation accuracy when applying various backbone models as the LCA-GAN generator. The last row in Table 8 is the method that subtracts the original image and masked image, concatenates the input image with the image where only the occluded area remains, and uses this input image in Pix2pix learning. Evidently, the best age estimation performance was achieved when using the Pix2pix backbone generator in LCA-GAN. In addition, we performed additional comparisons of the accuracy of the state-of-the-art age estimation method in Table 8. As shown in Table 8, the MAEs with original non-occluded and mask-occluded face images by the state-of-the-art age estimation method were 5.80 years (baseline 1) and 10.45 years (baseline 2) years, respectively. Although the MAE by our LCA-GAN is 6.64 years, it is much lower than that with mask-occluded face images without our LCA-GAN (baseline 2), which confirms the effectiveness of our proposed LCA-GAN.

Figure 11 shows examples of the generated images according to the ablation study in Table 8. In Table 8, the age estimation performance of baseline 1 using non-occluded facial images and baseline 2 using mask-occluded facial images (occluding nose and mouth) were 5.80 and 10.45, respectively, which is a difference of 4.65 years, indicating that the nose and mouth have important information for age estimation. In addition, the areas in the facial image that have important information for age estimation are shown, with significant activation in the nose and mouth. Figure 11a,b show the masked images and original images, and Figure 11c–f display the de-occluded images processed in the order in

Table 9. Comparative accuracies of age estimation by LCA-GAN and various de-occlusion methods (MPRNet* is a two-stage model that reduces one stage using the input image with the smallest size in MPRNet) (unit: years).

	LCA-GAN	AFD-StackGAN [54]	CFR-GAN [55]	MPRNet [83]	MPRNet* [83]	Pix2pix [57]	CycleGAN [84]
MAE	6.64	6.92	7.13	6.95	7.83	7.72	8.18

. According to the results in Figure 11, the proposed LCA-GAN using only Pix2pix yielded the best de-occlusion performance. The de-occlusion networks including Pix2pix and CycleGAN used in Table 8 and Figure 11, except for U-net, are adversarial networks using U-net as a generator. These adversarial learning-based de-occlusion methods generate robust and realistic de-occluded facial images by capturing complex patterns, but age information is somewhat lost due to unnecessary deformation in non-occluded areas of the eyes. On the other hand, CNN-based U-net is trained with a pixel-wise loss function and has less deformation in the non-occluded facial area of the eyes, so it preserves age information well. However, learning the mapping from input image to target image in the mask-occluded area is difficult and generates blurred images. Consequently, adversarial learning-based methods are weak at preserving information in non-occluded areas of the eyes and are strong at de-occlusion, while U-net is strong at preserving information in non-occluded areas of the eyes and is weak at de-occlusion. The areas with significant age information are shown in the human face image. The areas with high activation are the eyes, nose, and mouth, where the eyes are a non-occluded area and the nose and mouth are occluded areas. As a result, U-net preserved the age information in the non-occluded area of the eyes well, but the non-occluded areas were smaller than the whole face area, and the consequent de-occlusion performance was lower than the other methods because it cannot restore the age information.

Comparisons of our LCA-GAN with Existing Methods

This subsection compares the proposed method with state-of-the-art methods. MPRNet is a three-stage model with an iterative structure that receives input images through the multi-scale approach [83]. MPRNet* in Table 9 is a two-stage model that reduces one stage using the input image with the smallest size in MPRNet. According to the experimental results listed in Table 9, LCA-GAN, the de-occlusion network proposed in this study, yielded the best age estimation performance.

Figure 12 shows examples of mask de-occluded images obtained using the proposed LCA-GAN and state-of-the-art methods. Figure 12a displays masked facial images created by the method described in Section 4.1, and Figure 12b shows the original facial images. De-occluded images are shown by (c) the proposed LCA-GAN, (d) AFD-Stack GAN, (e) CFR-GAN, (f) MRPNet, (g) CycleGAN, and (h) Pix2pix. As shown in Figure 12, the mask de-occluded image generated by LCA-GAN is the nearest to the original image.

4.4. Testing with PAL Database

4.4.1. Comparisons of the Quality of Images Generated by Proposed Method and the State-of-the-Art Methods

We performed additional experiments using the open database PAL to confirm the generality of the proposed LCA-GAN performance. As presented in

Table 10. Comparative SSIM and PSNR of original image and de-occluded images.

	LCA-GAN	AFD-StackGAN [54]	CFR-GAN [55]	MPRNet [83]	Pix2pix [57]	CycleGAN [84]
SSIM	0.7042	0.6983	0.7207	0.7002	0.7134	0.6892
PSNR	18.3302	19.4423	18.3043	18.9742	19.4211	17.9443

, CFR-GAN [55] and AFD-StackGAN [54] exhibited the best performance for SSIM and PSNR, respectively, while the proposed LCA-GAN yielded the third and fourth highest performance for SSIM and PSNR, respectively. However, SSIM and PSNR are values that represent image quality according to de-occlusion, but the primary goal of this study is to improve age estimation accuracy (not image quality) through de-occlusion. According to a comparison of age estimation accuracy in

Table 11. Comparative accuracies of age estimation by LCA-GAN and various de-occlusion methods (MPRNet* is a two-stage model that reduces one stage using the input image with the smallest size in MPRNet) (unit: years).

	LCA-GAN	AFD-StackGAN [54]	CFR-GAN [55]	MPRNet [83]	MPRNet* [83]	Pix2pix [57]	CycleGAN [84]
MAE	6.12	6.94	6.52	8.21	8.70	7.12	9.02

, the proposed LCA-GAN yielded the highest accuracy.

4.4.2. Comparisons of Age Estimation Accuracy by Our LCA-GAN and the Existing Methods

For the next experiment, we de-occluded images using LCA-GAN and compared the age estimation accuracy using DEX. MPRNet* in Table 9 is a two-stage model that reduces one stage using the input image with the smallest size in MPRNet. As shown in Table 11, LCA-GAN, the de-occlusion network proposed in this study, yielded the best age estimation performance. In addition, we performed additional experiments using different age estimation methods after the same use of LCA-GAN. As shown in

Table 12. Comparative accuracies of different age estimation methods after the same use of LCA-GAN (unit: years).

	VGG-16 [68]	ResNet-50 [87]	ResNet-152 [87]	DEX [8]	AgeNet [11,88]	Inception with Random Forest [20]
MAE	6.20	7.22	6.32	6.12	6.19	6.42

, DEX showed the best accuracy among all the different age estimation methods.

Figure 13 illustrates examples of mask-de-occluded images obtained by the proposed LCA-GAN and the state-of-the-art methods. As evidenced in Figure 13, the mask de-occluded image generated by LCA-GAN is the closest to the original image.

4.5. Processing Speed

In this subsection, we measured and compared the processing times of the proposed LCA-GAN and state-of-the-art methods in the desktop environment described in Section 4.1 and a Jetson TX2 board [89], as shown in Figure 14.

Table 13. Comparative average processing time of one image by LCA-GAN and state-of-the-art methods. (unit: ms).

	Desktop Computer	Jetson TX2 Board
LCA-GAN	11.94	177.52
AFD-Stack GAN [54]	23.03	342.8
CFR-GAN [55]	35.6	541.5
MPRNet [83]	21.12	318.02
MPRNet* [83]	14.04	212.24
Pix2pix [57]	11.2	171.5
Cycle GAN [84]	23.2	353.1

lists the measured processing times. LCA-GAN yielded a faster processing speed in the desktop environment and embedded environment than all state-of-the-art methods, except Pix2pix [57]. Furthermore, the processing speed did not greatly differ from Pix2pix [57]. This indicates that the proposed LCA-GAN can be operated even in an embedded system environment with limited computing resources.

Table 14. Comparative model complexities of LCA-GAN and state-of-the-art methods.

	Number of Parameters	GFLOPs	Memory Usage (GB)
LCA-GAN	57,118,684	1.4668	0.5913
AFD-Stack GAN [54]	102,325,149	2.6764	0.5961
CFR-GAN [55]	171,588,876	2.8730	1.0925
MPRNet [83]	102,725,856	9.1092	4.5598
MPRNet* [83]	68,114,344	6.0728	3.0399
Pix2pix [57]	57,196,292	0.972	0.2062
Cycle GAN [84]	114,392,584	1.944	0.4124

compares the number of parameters, giga floating point operations per second (GFLOPs), and memory usage between the proposed LCA-GAN and state-of-the-art methods. LCA-GAN exhibited the smallest number of parameters, second lowest GFLOPs, and third lowest memory usage compared to the state-of-the-art methods. However, as indicated in Table 9 and Table 11, the proposed LCA-GAN yielded the best age estimation performance compared to the previous methods.

4.6. Discussion

In this subsection, we present the extraction and analysis of the attention map of the attention module used in the LCA-GAN de-occlusion process (Figure 15) and the gradient class activation map (Grad-CAM) [90] of DEX used for age estimation (Figure 16). Figure 15 displays the (a) original facial images and (b) the mask-occluded facial images. Figure 15c–e show the attention maps of LCAB 1, LCAB 3, and LCAB 5 of the Table 3 encoder, respectively, and Figure 15f–h show the attention maps of LCAB 6, LCAB 8, and LCAB 10 of the Table 3 decoder, respectively. The attention maps show that in the proposed LCA-GAN, as the encoder de-occlusion progresses, attention is activated from the entire face area to the mask area. Subsequently, as the decoder de-occlusion progresses, high activation is shown in the detailed areas of major facial elements, such as the eyes, nose, mouth, and chin.

Subsequently, we examined the Grad-CAM images of DEX, the age estimation network used in this experiment. Figure 16 shows (a) the original images, (b) the mask-occluded images, and (c) the de-occluded images. In Figure 15d–g, the Grad-CAM images of DEX’s 4th, 8th, and 11th convolution layers and the last max pooling layers are overlapped with the mask de-occluded images. As illustrated in Figure 16, as DEX learns to estimate the age from mask de-occluded facial images, in Grad-CAM, which showed high activation for high-frequency information in large areas of the image, elements such as the eyes, nose, mouth, and the surrounding textures are activated with the deepening of layers.

Figure 17 shows examples of de-occluded images that were incorrectly generated by the proposed LCA-GAN. These incorrectly generated de-occluded images can be attributed to several problems: first, the difference in convergence speed between the generator and discriminator during adversarial learning; second, the use of a single generator; and finally, the class imbalance according to gender and race in the learning images as well as non-uniform lighting in the test images.

5. Conclusions

Mask-occluded images that occur in real environments cause a loss of information required for age estimation, thereby degrading age estimation performance. This study proposed a novel de-occlusion network LCA-GAN. Through experiments using MORPH and PAL, open databases of human facial images, the proposed network achieved higher age estimation performance than existing state-of-the-art de-occlusion networks. Furthermore, the proposed LCA-GAN contains 57,118,684 parameters, which is fewer than existing methods. This indicates that it can be operated even in an embedded system with limited computing resources. Moreover, from the attention maps in LCA-GAN and Grad-CAM images of DEX for images de-occluded with LCA-GAN, LCA-GAN and DEX effectively extracted features for de-occlusion and age estimation, respectively. However, as shown in Figure 17, LCA-GAN occasionally incorrectly generated de-occluded images.

To solve this, it is necessary to research solutions for several problems: the difference in convergence speed between the generator and discriminator during adversarial learning, the use of a single generator, the class imbalance according to gender and race in the learning images, and non-uniform lighting in the test images. Moreover, we will investigate solutions for cases where mask occlusion simultaneously occurs with other factors, such as low light and image blurring. Furthermore, we will research a shallower model to achieve faster processing speeds in an embedded platform.