Facial Beauty Prediction Using an Ensemble of Deep Convolutional Neural Networks ^†

Abstract

The topic of facial beauty analysis has emerged as a crucial and fascinating subject of human culture. With various applications and significant attention from researchers, recent studies have investigated the relationship between facial features and age, emotions, and other factors using multidisciplinary approaches. Facial beauty prediction is a significant visual recognition problem in the assessment of facial attractiveness, which is consistent with human perception. Overcoming the challenges associated with facial beauty prediction requires considerable effort due to the field’s novelty and lack of resources. In this vein, a deep learning method has recently demonstrated remarkable abilities in feature representation and analysis. Accordingly, this paper proposes an ensemble based on pre-trained convolutional neural network models to identify scores for facial beauty prediction. These ensembles are three separate deep convolutional neural networks, each with a unique structural representation built by previously trained models from Inceptionv3, Mobilenetv2, and a new simple network based on Convolutional Neural Networks (CNNs) for facial beauty prediction problems. According to the SCUT-FBP5500 benchmark dataset, the obtained 0.9350 Pearson coefficient experimental result demonstrated that using this ensemble of deep networks leads to a better prediction of facial beauty closer to human evaluation than conventional technology that spreads facial beauty. Finally, potential research directions are suggested for future research on facial beauty prediction.

1. Introduction

The human face holds a distinct importance in our social interactions, and the pursuit of beauty, particularly facial beauty, is an enduring and ubiquitous feature of human nature. In recent years, there has been a significant increase in the demand for aesthetic surgery, underscoring the importance of a nuanced understanding of beauty in medical settings [1]. Remarkably, the exploration of physical beauty in humans has a storied history dating back over 4000 years, demonstrating the enduring relevance of this topic [2]. The importance of physical beauty in the face has been studied for hundreds of years, and its influence on social decisions such as partner choices and hiring decisions is well documented [3]. The perception of facial attractiveness is considered a highly desirable physical trait, with philosophers, artists, and scientists attempting to understand the secrets of beauty [4].

Facial beauty prediction is an emerging topic that is receiving increasing attention from researchers and users alike, particularly in the field of facial recognition and understanding [5]. Beauty is viewed as a form of information in computer-based face analysis, and it is linked to how people perceive attractiveness. In the field of psychology, several theories have been established on how people observe facial attractiveness. However, studying face attractiveness using computers is a relatively recent research area, with limited resources and few articles published on this subject. Several works have focused on analysing the irregular features of facial attractiveness [6].

The analysis of facial attractiveness presents two main challenges. First, the complexity of human perception and the wide variety of facial features make it difficult to build robust and effective models for evaluating beauty. Secondly, many face reference databases are primarily configured for face recognition problems and are not suitable for attractiveness prediction [7]. Therefore, most facial beauty studies focus on designing facial beauty descriptors [8].

In recent years, most research on facial beauty prediction has been based on deep learning methods [9]. The development of deep learning architecture has been driven by the strength and adaptability of these algorithms, particularly convolutional neural networks (CNNs) [10]. These algorithms offer a novel perspective on the facial beauty prediction problem and have shown promising results for several computer vision applications, such as face recognition, object identification, semantic segmentation, image classification, biomedical analysis, captioning, and biometrics [11]. DCNNs perform much better [12]. We created new ensemble models for facial attractiveness evaluation. As a result, this study proposes ensembles as three separate deep convolutional neural networks for the facial beauty prediction (FBP) problem. The contributions of this paper are as follows:

• The investigation of the effectiveness of conventional transfer learning techniques for facial beauty prediction.
• We provide an ensemble regression for facial attractiveness evaluation using the projected scores of networks with with InceptionV3 and MobileNetV2, and a new simple network based on Convolutional Neural Networks is proposed with loss functions.
• The efficiency of the suggested approach is demonstrated using the specialized FBP dataset, SCUT-FBP-5500. The efficiency of merging the assessments of several predictors in the proposed ensemble DCNNs regression model, which is considerably compatible with the ground truth of the dataset used, is demonstrated by the findings, which are encouraging. We have made our scripts and pre-processed pictures available to the general public at (https://github.com/DjameleddineBoukhari/ENCNN, accessed on 28 January 2023).

The structure of this paper is as follows: Some related research on facial attractiveness prediction is included in Section 2. Section 3 explains the selection process of the architectures used. On the SCUT-FBP5500 dataset, Section 4 provides the experimental findings and performance assessments.

2. Convolution Neural Networks Architecture for FBP

The initial concept of neural networks was extended by the human nervous system by mimicking the human nervous system. Scientists propose the concept of neural networks. Convolutional neural networks are further improvements of the neural network concept. The arrival of this model is good news for auto-vision [13].

Numerous techniques based on deep neural networks (DNN) have been developed for FBP. One of the most popular CNN architectures, ResNet [14], has been utilized in a number of computer vision applications.

K. Cao et al. [5] used residual-in-residual (RIR) groups to build a deeper network. A combined spatial-wise and channel-wise attention mechanism was introduced for better feature comprehension. The authors presented their facial beauty database SCUT-FBP5500 [15], with two evaluation protocols (five-fold cross-validation 80–20% and 60–40% split). They tested three CNN architectures, namely Alexnet [10], Resnet-18 [14], and ResneXt-50 [14]. The results show better feature comprehension. In [16], R3CNN architecture is proposed to integrate relative ranking into regression to improve the performance of FBP, and it can be flexibly implemented using existing CNNs as a backbone network. This architecture provides better results than the SCUT-FBP [17] and SCUT-FBP5500 [15] datasets.

3. Methodology

In order to predict facial beauty, this work builds an ensemble of trained models. Mainly, our proposed EN-CNN architecture approach focuses on three pre-trained models, and the estimates of both are combined to create a final prediction of facial beauty [18]. Transfer learning is used by the pre-trained models to reduce their weights so that they can perform a comparable regression task. For facial beauty, an ensemble learning of trained models achieves greater performance. Consequently, in this study, we transfer the weights of a set of three potent pre-trained CNN models. The next part presents the planned ensemble learning as well as the pre-trained Deep CNN models [19].

3.1. Pre-Trained InceptionV3

The Inception V3 [20] CNN was introduced by Google teams in [11]. Based on the Inception V1 model, Inception V3’s architecture was updated. The InceptionV3’s design incorporates numerous kernel types at the same level. Instead of using a large filter size of 7 × 7 and 5 × 5, the InceptionV3 uses a modest filter size of 1 × 7 and 1 × 5. Furthermore, a bottleneck of 1 × 1 convolutions is used, improving feature representation as a result. Beginning with the input data, three distinct convolutional layers with 3 × 3 or 5 × 5 filter sizes are created by mapping parallel calculations. These layers’ output is combined into one layer, which is known as the output layer.

3.2. Pre-Trained MobileNetV2

A lightweight CNN model called MobileNet [21] is built on inverted residuals and a linear bottleneck, which provide quick connections between thin layers. It is a low-latency model that uses a small amount of power; therefore, it can be used to manage limited hardware resources. The MobileNet’s key benefit is the trade-off it makes between several elements including latency, accuracy, and resolution. In MobileNet, feature maps are generated using point-wise convolutional kernels and depth separable convolutional (DSC) kernels. In MobileNet [21], DSC first filters the input image’s spatial dimensions using depth-wise kennel 2-D filters. The depth-wise filter has a size of Dk × Dk × 1, which is significantly smaller than the size of the input images [22].

3.3. S-CNNs Network

In order to predict facial beauty, this work builds simple CNNs. Mainly, our proposed S-CNN architecture approach is constructed of several convolution layers and one fully connected layer at the end. Within each convolution layer, a 2D convolution is carried out, followed by ReLU activation. To demonstrate the effectiveness of the present S-CNNs, we control the recent progress in neural architecture search to develop a new family of MixConv-based models. Our neural architecture is an ensemble of simple CNNs models, where the contribution resides in the method of layer mixing with a kernel size of 3 × 3, 5 × 5, illustrated in Figure 1.

3.4. The Proposed EN-CNNs

The proposed ensemble of deep CNNs (EN-CNNs) architecture follows the three previously trained models (InceptionV3, MobileNetV2, and S-CNN). In the proposed EN-CNNs for the automated classification system, they act as basic classifiers of facial beauty. The details of the proposed EN-CNNs are depicted in Figure 2.

4. Experiment

This section describes the experiments and assessment findings from the models (EN-CNN) used in this study. The SCUT-FBP5500 dataset [15] is used for network training. Our network is trained for 200 iterations with a batch size of b = 25. The Adam optimizer updates the parameters with a learning rate of lr = 1e − 6. The selected loss function was MSE.

The SCUT-FBP5500 Dataset

The standard SCUT-FBP5500 dataset [15] is introduced in this study, which comprises 5500 frontal face images at 350 × 350 resolutions with various attributes, including race (Asian/Caucasian), gender (female/male), and age (15–60) [23,24].

As shown in Figure 3, the ground truth rating for each face in the dataset is the average of all evaluations given on a scale from 1 to 5 by the 60 ratters. This enables the use of various computational models with various facial attractiveness prediction paradigms. The 2000 Asian females (AF), 2000 Asian men (AM), 750 Caucasian females (CF), and 750 Caucasian males (CM) are the four subsets of the SCUT-FBP5500 Dataset that may be separated according to race and gender [25].

5. Results and Discussion

An automatic face attractiveness estimate is achieved using an ensemble DCNNs regression. The experimental findings from the suggested EN-CNN are presented in this subsection. We carry out comparisons utilizing a range of techniques, including geometric feature-based and deep learning-based techniques, such as AlexNet, ResNet-18, ResNeXt-50, CNN–SCA, R3CNN, and Semi-supervised, etc. MAE, RMSE, and PC are chosen as the metrics.

Table 1. Performance comparisons of the SCUT-FBP5500 dataset.

Methods	Pre-Training	MAE ↓	RMSE ↓	PC ↑
AlexNet [15]	ImageNet	0.2651	0.3481	0.8634
ResNet-18 [15]	ImageNet	0.2419	0.3166	0.8900
ResNeXt-50 [15]	ImageNet	0.2291	0.3017	0.8997
CNN–SCA [5]	ImageNet	0.2287	0.3014	0.9003
R3CNN [16]	ImageNet	0.2120	0.2800	0.9142
Semi-supervised [23]	VGGFace2	0.2210	0.2870	0.9113
CNN-ER [26]	VGGFace2	0.2009	0.2650	0.9250
NAS4FBP Net [27]	ImageNet	0.1939	0.2579	0.9275
EN-CNN Ours	ImageNet	0.1933	0.2482	0.9350

shows the performance comparisons of the SCUT-FBP5500 dataset of facial beauty prediction to testify the EN-CNN capacity via comparison, which holds 80–20% splitting.

Typically, the quantity of parameters imposes a limit on the performance improvement. The present model performs better than other models (AlexNet, ResNet-18, ResNeXt-50, CNN–SCA, R3CNN, and Semi-supervised, etc.). It uses an ensemble of deep CNNs (EN-CNNs) architectures for the three models. Our EN-CNNs hold 26.99 M parameters, InceptionV3 with 22.85 M parameters model, MobileNetV2 with 2.91 M parameters model, and our S-CNN network holds 1.23 M parameters and 224.16 MFlops. CNN–SCA has 6.75 M parameters and 34.25 BFlops. ResNeXt-50 has 25.03 M parameters and 5.56 BFlops. AlexNet has 62.38 M parameters and 1. 5 BFlops. This comparison reveals that our network is better than those of the cited works. This confirms that both the proposed EN-CNNs network played a crucial role in outperforming the state-of-the-art methods. In Figure 4, we present the comparisons of the predicted scores.

6. Conclusions

In this paper, an ensemble of deep CNNs for facial beauty prediction is proposed. We present an ensemble regression for facial beauty estimation by studying the effect of the standard transfer learning approaches on the facial beauty prediction problem and by combining the predicted scores of networks with a three-branch network (InceptionV3, MobileNetV2, and S-CNN) trained with loss functions. We describe and optimize a set of hyper-parameters for the new set of pre-trained models to classify facial beauty. By utilizing an ensemble of all the previously mentioned transfer learning techniques, an ensemble (EN-CNN) was developed to predict scores in facial beauty. The experimental findings show that our network can perform better than the previous CNN baseline approaches. The experimental results showed that the proposed network achieved better performance compared to several works available in the available literature. It improves the assessment’s congruence with human judgment. From this perspective, we propose to expand the scope of the database and improve the network using different architectures collected from Transfomer and ResNeSt.