A Reference-Guided Double Pipeline Face Image Completion Network

Abstract

The existing face image completion approaches cannot be utilized to rationally complete damaged face images where their identity information is completely lost due to being obscured by center masks. Hence, in this paper, a reference-guided double-pipeline face image completion network (RG-DP-FICN) is designed within the framework of the generative adversarial network (GAN) completing the identity information of damaged images utilizing reference images with the same identity as damaged images. To reasonably integrate the identity information of reference images into completed images, the reference image is decoupled into identity features (e.g., the contour of eyes, eyebrows, nose) and pose features (e.g., the orientation of face and the positions of the facial features), and then the resulting identity features are fused with posture features of damaged images. Specifically, a lightweight identity predictor is used to extract the pose features; an identity extraction module is designed to compress and globally extract the identity features of the reference images, and an identity transfer module is proposed to effectively fuse identity and pose features by performing identity rendering on different receptive fields. Furthermore, quantitative and qualitative evaluations are conducted on a public dataset CelebA-HQ. Compared to the state-of-the-art methods, the evaluation metrics peak signal-to-noise ratio (PSNR), structure similarity index (SSIM) and L1 loss are improved by 2.22 dB, 0.033 and 0.79%, respectively. The results indicate that RG-DP-FICN can generate completed images with reasonable identity, with superior completion effect compared to existing completion approaches.

1. Introduction

Face image completion has been extensively utilized as a research hotspot in the field of computer vision in the film industry and detection. Face image completion should effectively restore the reasonable identity of a damaged image, i.e., ensure that the completed image has the same identity as the original image. Mainly, traditional image completion approaches are based on patches or diffusion. The diffusion-based approach [1,2] is to iteratively propagate the low-level features from known regions to unknown areas along the mask boundaries to complete the small-hole area with weak structure. The patch-based approach [3,4,5,6,7,8] is to search for similar patches in the same dataset or image to complete the unknown areas. Both approaches assume unknown areas share similar content with known regions; thus they can directly match, copy and realign the known patches to complete unknown areas, but cannot predict unique content not present in the known regions. Thus, these methods are only appropriate for filling natural images with similar textures. Nevertheless, face images have stronger topolopy than natural images, hence, they cannot be completed by such traditional approaches.

Presently, the approach based on deep learning and a generative adversarial network (GAN) [9] has attracted research on face image completion. In such an approach, a trainable completion network is made mostly based on a convolutional neural network, and the completion network is utilized as a discriminator and generator for adversarial training to generate natural and real completed images. A context encoder was proposed by Pathak et al. [10] using the generative adversarial loss for training, based on which Iizuka et al. [11] proposed local and global discriminant loss. Yu et al. [12] put forward a completion plan of rough completion first and then fine completion to obtain a high-definition completion effect through long-range feature transfer using the semantic attention layer. Moreover, some scholars proposed partial convolution [13] and gated convolution [14] to enhance the network robustness to the mask shape. Given the problem of a single completion result, Zhang et al. [15] mapped the damaged images to probability distribution utilizing a modified variational encoder and GAN, so that multiple completion results can be constructed for one damaged image by sampling the probability distribution.

The completion results of the dataset-guided face image completion scheme [10,11,12,13,14,15] are natural and real on face images with completely lost identity information. However, the results are essentially average faces based on the entire dataset, due to their identities being uncertain. Therefore, Li et al. [16] proposed semantic parsing loss, effectively enhancing the similarity between the completion result and the topological structure of the original images. Some scholars [17,18,19,20,21] used landmarks and edges as prior knowledge to guide the completion process. Compared with the approach of [16], the visual quality of the completion results of such approaches is greatly improved with more stable training. However, these approaches can only provide a posture information guide to the completion procedure, and the completion results still have no certain identity.

To solve the identity uncertainty problem, a reference-guided double-pipeline face image completion network (RG-DP-FICN) is proposed in this paper to guide the entire completion process using the identity information of reference images. The aim of the network is to transfer the identity information from the reference images to completed images, thereby improving the identity plausibility of the completed images.

The key points of this paper are as follows:

(1)

The reference-guided completion network is used to determine the rationality of the identity of completion results.

(2)

A double-pipeline GAN is proposed to realize the decoupling and fusion of identity and posture features.

(3)

An attention feature fusion module is designed to restore the background information lost during fusion.

2. Materials and Methods

2.1. Dataset and Evaluation Metrics

All the experiments conducted in this paper exploit a high-quality human face dataset CelebA-HQ [22]. This dataset contains 30,000 high-quality face images coming from 6217 identities. It is trained and tested with the processing as follows. First, the images with the same identity in the dataset are summarized to obtain 6217 image lists. Each list contains a different number of face images. Then, the lists of only one image with the same identity are eliminated. Two images are selected in order from the list as the original image and the reference image, respectively, and 28,299 image pairs are obtained. Ultimately, 27,299 image pairs are chosen as the training set, while the remaining 1000 image pairs as the test set. All images are scaled to 256 × 256 × 3.

The evaluation metrics include peak signal-to-noise ratio (PSNR) [23], structure similarity index (SSIM) [23] and L1 loss. These metrics are used to measure the pixel-level difference and overall similarity between the completed image and the original image. For PSNR and SSIM, higher values indicate better performance, while for L1 loss, the lower the better. The formulas for the calculation of these performance metrics are as follows:

$$\begin{array}{cc}\hfill & MSE=\frac{1}{H\times W}\times \sum _{i=1}^H\sum _{j=1}^W{({\mathit{I}}_{\mathit{g}}(i,j)-{\mathit{I}}_{\mathit{s}}(i,j))}^2\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & PSNR=10\times lo{g}_{10}\left(\frac{{(2^n-1)}^2}{MSE}\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & SSIM=l\times c\times s,l=\frac{2\times {\mu }_g\times {\mu }_s+C_1}{{\mu }_g^2+{\mu }_s^2+C_1},c=\frac{2\times {\sigma }_g\times {\sigma }_s+C_2}{{\sigma }_g^2+{\sigma }_s^2+C_2},s=\frac{{\sigma }_{gs}+C_3}{{\sigma }_g^2+{\sigma }_s+C_3}\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & L1loss=\frac{1}{H\times W}\times \sum _{i=1}^H\sum _{j=1}^W\left|I_g(i,j)-I_s(i,j)\right|\hfill \end{array}$$

(4)

where, ${\mathit{I}}_{\mathit{s}}$ represents the original image, ${\mathit{I}}_{\mathit{g}}$ represents the completed image, i and j represent the coordinate positions of pixels in ${\mathit{I}}_{\mathit{s}}$ and ${\mathit{I}}_{\mathit{g}}$; H, W represent the width and height of the image respectively; l, c and s represent the image similarity of ${\mathit{I}}_{\mathit{s}}$ and ${\mathit{I}}_{\mathit{g}}$ in terms of luminance, contrast, and structure, respectively; ${\mu }_s$, ${\mu }_g$${\sigma }_s$, ${\sigma }_g$ represent the mean and variance of ${\mathit{I}}_{\mathit{s}}$ and ${\mathit{I}}_{\mathit{g}}$, ${\sigma }_{gs}$ represent the covariance of ${\mathit{I}}_{\mathit{s}}$ and ${\mathit{I}}_{\mathit{g}}$; $C1$, $C2$, $C3$ represent the constant in order to avoid the denominator is 0, generally take $C1={(K1\times L)}^2$, $C2={(K2\times L)}^2$, $C3={(K3\times L)}^2$, $K1=0.01$, $K2=0.03$, $L=255$.

2.2. Double-Pipeline GAN

Considering identity uncertainty problem in the completed image, reference image ${\mathit{I}}_{\mathit{a}}$ with the same identity as the damaged image ${\mathit{I}}_{\mathit{m}}$ is incorporated to guide the completion procedure. In this study, a reference guided double-pipeline face image completion network(RG-DP-FICN) is proposed (Figure 1). Two pipelines (reconstruction and completion) are constructed in this network, to reasonably incorporate the identity information of reference images. To realize the identity transfer from reference image to damaged image, the posture features(e.g the orientation of face and the positions of the facial features) and identity features(e.g., the contour of eyes, eyebrows, nose) of reference images are decoupled through such double-pipeline network design, and the identity features of reference images are fused with the posture features of damaged images as follows.

The reconstruction pipeline possesses all the information on reference images and the information is jointly obtained by the encoder E and identity encoder $E_{id}$. The reconstruction process is as follows: First, the damaged image ${\mathit{I}}_{\mathit{ma}}$ and the topological structure of the reference image ${\mathit{L}}_{\mathit{a}}$ are encoded via the encoder E to attain posture feature ${\mathit{z}}_{\mathit{ma}}$. Then, ${\mathit{z}}_{\mathit{ma}}$ and identity feature ${\mathit{z}}_{\mathit{id}}$ extracted by $E_{id}$ are fused utilizing the identity transfer module to obtain ${\mathit{z}}_{\mathit{ua}}$. Ultimately, ${\mathit{z}}_{\mathit{ua}}$ is inserted into the decoder F for decoding to attain the reconstructed image ${\mathit{I}}_{\mathit{rec}}$ (Equations (5) and (6)). Therefore, by supervising the similarity between the reconstructed image ${\mathit{I}}_{\mathit{rec}}$ generated via the pipeline, the reference image ${\mathit{I}}_{\mathit{a}}$ can effectively realize the decoupling of the posture feature ${\mathit{z}}_{\mathit{ma}}$ and the identity feature ${\mathit{z}}_{\mathit{id}}$. As for the completion pipeline, the damaged image ${\mathit{I}}_{\mathit{m}}$ and the topological structure ${\mathit{L}}_{\mathit{g}}$ are encoded by the encoder E to obtain ${\mathit{z}}_{\mathit{m}}$. Then, the identity feature ${\mathit{z}}_{\mathit{id}}$ obtained by decoupling the reference image and the posture feature ${\mathit{z}}_{\mathit{m}}$ of the damaged image are fused utilizing the identity transfer module to obtain ${\mathit{z}}_{\mathit{u}}$. Ultimately, ${\mathit{z}}_{\mathit{ua}}$ is inserted into the decoder F for decoding to generate the completed image ${\mathit{I}}_{\mathit{g}}$ with certain identity (Equations (7) and (8)).

$$\begin{array}{cc}\hfill & {\mathit{I}}_{\mathit{rec}}=G({\mathit{I}}_{\mathit{ma}},{\mathit{L}}_{\mathit{a}},E_{id}\left({\mathit{I}}_{\mathit{a}}\right))\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & {\mathit{I}}_{\mathit{ma}}={\mathit{I}}_{\mathit{a}}\times \mathit{M}\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & {\mathit{I}}_{\mathit{g}}=G({\mathit{I}}_{\mathit{m}},{\mathit{L}}_{\mathit{g}},E_{id}\left({\mathit{I}}_{\mathit{a}}\right))\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & {\mathit{I}}_{\mathit{m}}={\mathit{I}}_{\mathit{s}}\times \mathit{M}\hfill \end{array}$$

(8)

where, ${\mathit{I}}_{\mathit{s}}$ represents the original image; $\mathit{M}$ is a mask; ${\mathit{L}}_{\mathit{a}}$ and ${\mathit{L}}_{\mathit{g}}$ denote the face topological structure obtained by connecting the landmarks of ${\mathit{I}}_{\mathit{a}}$ and ${\mathit{I}}_{\mathit{m}}$ predicted by the landmark predictor [20].

The generator G is trained to enhance the visual quality of ${\mathit{I}}_{\mathit{g}}$ and the rationality of its topological structure through adversarial training with two discriminators $D_1$ and $D_2$. For $D_1$, the true sample and the false sample are the image pair $({\mathit{L}}_{\mathit{g}},{\mathit{I}}_{\mathit{s}})$ and $({\mathit{L}}_{\mathit{g}},{\mathit{I}}_{\mathit{g}})$, respectively. For $D_2$, the true sample and the false sample are respectively the image pair $({\mathit{L}}_{\mathit{a}},{\mathit{I}}_{\mathit{a}})$ and $({\mathit{L}}_{\mathit{a}},{\mathit{I}}_{\mathit{rec}})$. Such sample pair design can effectively enhance the image quality of ${\mathit{I}}_{\mathit{g}}$ and ${\mathit{I}}_{\mathit{rec}}$, and ensure the rationality of their topological structure.

2.3. Identity Encoder and Identity Transfer Module

The fusion of the identity information of reference images and the original information of damaged images is a challenging problem. This is because first, the reference images present different visual appearances due to gender, lighting conditions, and makeup, hence, it is difficult to obtain identity information. Second, there are differences in the face posture along with different proportions and positions of facial features in the image, hence, it is difficult to rationally fuse identity information. Hence, to solve this problem, an identity transfer module and an identity encoder are designed in this paper (Figure 2) to extract the identity information and feature fusion, respectively. Next, the transfer process in the completion pipeline is introduced as an example.

In the identity encoder, the residual block and the self-attention module [24] are used first to extract and compress identity information of reference image (256 × 256 × 3) to obtain a feature (512 × 4 × 4). Then, the feature is mapped into the identity feature ${\mathit{z}}_{\mathit{id}}$ (512 × 1 × 1) through a fully connected layer. As a global operation, each unit of the identity feature ${\mathit{z}}_{\mathit{id}}$ is the weighted sum of 512 × 4 × 4 feature values, which allows each unit in ${\mathit{z}}_{\mathit{id}}$ to reason about reference image identity. Hence, ${\mathit{z}}_{\mathit{id}}$ could well represent the identity of reference images.

In the identity transfer module, ${\mathit{z}}_{\mathit{m}}$ and ${\mathit{z}}_{\mathit{am}}$, which representing high-dimensional semantics, is globally adjusted step by step using the style rendering block to realize identity transfer. The Identity transfer module consists of four style rendering blocks, each of which contains two AdaIn layers. The specific structure of style rendering block is represented in Figure 3, which is employed twice through adaptation instance normalization (AdaIN) [25], as follows:

$$\begin{array}{c}\hfill {\mathit{f}}_{\mathit{i},\mathit{j}}^{\ast }=\frac{{\mathit{f}}_{\mathit{i},\mathit{j}}-u\left({\mathit{f}}_{\mathit{i},\mathit{j}}\right)}{\sigma \left({\mathit{f}}_{\mathit{i},\mathit{j}}\right)}\times {\mathit{\beta }}_{\mathit{i},\mathit{j}}+{\mathit{\alpha }}_{\mathit{i},\mathit{j}}\end{array}$$

(9)

where, ${\mathit{f}}_{\mathit{i},\mathit{j}}$ represents the input feature, $u\left({\mathit{f}}_{\mathit{i},\mathit{j}}\right)$ and $\sigma \left({\mathit{f}}_{\mathit{i},\mathit{j}}\right)$ denote the mean and variance of ${\mathit{f}}_{\mathit{i},\mathit{j}}$, respectively, $({\mathit{\alpha }}_{\mathit{i},\mathit{j}},{\mathit{\beta }}_{\mathit{i},\mathit{j}})$ represents the AdaIn affine parameter obtained by the fully connection layer mapping of identity feature ${\mathit{z}}_{\mathit{id}}$, ${\mathit{f}}_{\mathit{i},\mathit{j}}^{\ast }$ is the rendered feature, i indicates the i-th style rendering block, and j shows the j-th AdaIn of each style rendering block.

To improve the rendering effect, the receptive field is expanded utilizing the dilated convolution [26] in the style rendering blocks, thus, identity rendering in various receptive fields is achieved by continuously using the style rendering blocks. The entire style rendering process is completed by 4 style rendering blocks. The AdaIn affine parameters required by style rendering blocks are obtained by mapping the identity feature ${\mathit{z}}_{\mathit{id}}$ extracted by the identity encoder module. Thus, the adjusted code ${\mathit{z}}_{\mathit{u}}$ has rationality of identity. Similarly, the code ${\mathit{z}}_{\mathit{au}}$ of the reconstruction pipeline also possesses the reasonable identity, due to it uses the same identity transfer operation as completion pipeline.

2.4. Attention Feature Fusion Module

After identity transfer utilizing the identity transfer module, the codes ${\mathit{z}}_{\mathit{u}}$ and ${\mathit{z}}_{\mathit{au}}$ are obtained with reasonable identity. However, AdaIn in the identity transfer module will globally adjust the feature map during transfer, thus, it leads to the loss of background information. Therefore, an attention feature fusion module is designed in this paper to combine the undamaged features of background information before transfer and the code with reasonable identity to restore the background information of face images. For instance, ${\mathit{f}}_{\mathit{m}}$ and ${\mathit{f}}_{\mathit{u}}$ obtained by ${\mathit{z}}_{\mathit{u}}$ sampling are fused in the case of completion pipeline.

Figure 4 represents the structure of the attention feature fusion module. First, the attention score $\mathit{\delta }$ is calculated utilizing ${\mathit{f}}_{\mathit{u}}$ (Equation (10)). Then, ${\mathit{f}}_{\mathit{m}}$ and ${\mathit{f}}_{\mathit{u}}$ are fused under the guidance of $\mathit{\delta }$ (Equations (11)–(14)). Ultimately, ${\mathit{\gamma }}_{\mathit{m}}$ and ${\mathit{\gamma }}_{\mathit{u}}$ are spliced and then the fusion feature ${\mathit{f}}_{\mathit{g}}$ is obtained through a residual block(Equation (15)). Similarly, the fusion feature ${\mathit{f}}_{\mathit{rec}}$ is obtained by fusing ${\mathit{f}}_{\mathit{ma}}$ and ${\mathit{f}}_{\mathit{au}}$ using this module.

$$\begin{array}{cc}\hfill & {\delta }_{i,j}=\frac{exp\left(s_{i,j}\right)}{{\sum }_{i=1}^{N}exp\left(s_{i,j}\right)},\left(s_{i,j}\right)=P{\left(f_{ui}\right)}^{T}Q\left(f_{ui}\right)\hfill \end{array}$$

(10)

where, N denotes the number of pixels in ${\mathit{f}}_{\mathit{u}}$, and P and Q are $1\times 1$ convolutions.

$$\begin{array}{cc}\hfill & c_{mj}=\sum _{i=1}^N{\delta }_{ji}O\left(f_{mi}\right)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & c_{uj}=\sum _{i=1}^N{\delta }_{ji}O\left(f_{ui}\right)\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & {\mathit{y}}_{\mathit{m}}={\gamma }_m\times (1-\mathit{M})\times {\mathit{c}}_{\mathit{m}}+\mathit{M}\times {\mathit{f}}_{\mathit{m}}\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & {\mathit{y}}_{\mathit{u}}={\gamma }_u\times {\mathit{c}}_{\mathit{u}}+{\mathit{f}}_{\mathit{u}}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & {\mathit{f}}_{\mathit{g}}=S({\mathit{y}}_{\mathit{m}},{\mathit{y}}_{\mathit{u}})\hfill \end{array}$$

(15)

where, O represents a $1\times 1$ convolution, and γ_m & γ_u denote equilibrium parameters, S represents a residual block.

To verify the effectiveness of the attention feature fusion module, the additive operation is imposed for the multiple channels of the feature map ${\mathit{f}}_{\mathit{u}}$ and ${\mathit{f}}_{\mathit{g}}$ before and after fusion in this paper, for which the visualized results are represented in Figure 5. It is observed that the information before fusion is mainly concentrated in the face area, however, the background information is restored after fusion indicating that the module can effectively fuse the features.

2.5. Loss Function

According to the respective tasks of the generation pipeline and the reconstruction pipeline, in designing their loss function, it is considered that the reconstructed images should have a great similarity with the reference images, and the generated images should be real and natural with reasonable semantics.

First, reconstruction loss is introduced for the reconstruction pipeline to decrease the pixel-level difference between ${\mathit{I}}_{\mathit{rec}}$ and ${\mathit{I}}_{\mathit{a}}$. There is no complete face image information in the completion pipeline, hence, erroneous training results will be obtained when using the reconstruction loss forcibly. Therefore, spatially discounted reconstruction loss [12] is introduced, and the spatial discounted weight ${\mathit{W}}_{\mathit{sd}}$ is adjusted to apply a weaker pixel-level supervision to pixels farther from the nearest known pixel. The reconstruction loss and spatial reconstruction loss can be written as:

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{app}^r=\sum _i{∥{\mathit{I}}_{\mathit{rec}}-{\mathit{I}}_{\mathit{a}}∥}_1\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{app}^g=\sum _i{∥{\mathit{W}}_{\mathit{sd}}\times ({\mathit{I}}_{\mathit{g}}-{\mathit{I}}_{\mathit{s}})∥}_1\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & {\mathit{W}}_{\mathit{sd}}={\gamma }^l\hfill \end{array}$$

(18)

where, ${∥·∥}_1$ denotes the $L_1$ norm, ${\mathit{W}}_{\mathit{sd}}$ represents the weight of each pixel, l is the distance of the pixel to the nearest known pixel. $\gamma$ is set to 0.9 in all experiments.

Then, the perceptual loss [27] is introduced for the reconstruction pipeline to decrease the perceptually difference between ${\mathit{I}}_{\mathit{rec}}$ and ${\mathit{I}}_{\mathit{a}}$. The output distance of the reconstructed image ${\mathit{I}}_{\mathit{rec}}$ and the reference image ${\mathit{I}}_{\mathit{a}}$ at different layers of the pre-training network is penalized to enhance their perceptual similarity. The perceptual loss is determined utilizing the feature map ${\varphi }_i(·)$ output by the pre-trained VGG19 network on the ImageNet dataset [28], as follows:

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{pc}=\sum _i\frac{1}{N}\times {∥{\varphi }_i\left({\mathit{I}}_{\mathit{g}}\right)-{\varphi }_i\left({\mathit{I}}_{\mathit{s}}\right)∥}_1\hfill \end{array}$$

(19)

where, ${\varphi }_i\left({\mathit{I}}_{\mathit{g}}\right)$$ and ${\varphi }_i\left({\mathit{I}}_{\mathit{s}}\right)$ denote the feature map corresponding to ${\mathit{I}}_{\mathit{g}}$ and ${\mathit{I}}_{\mathit{s}}$, respectively.

Ultimately, to decrease the difference in data distribution between the generated image and the real image, and enhance the visual quality of the generated image, the discriminant loss is introduced for the two pipelines. The discriminant loss for the completion pipeline is designed based on LSGAN [29] (Eqtuion (20)), which could effectively improve the authenticity of the completed image ${\mathit{I}}_{\mathit{g}}$. Inspired by [30], the LSGAN-based discriminant loss is deformed (Eqation (21)) for the reconstruction pipeline. Such a design can encourage the original features $D_2({\mathit{I}}_{\mathit{a}},{\mathit{L}}_{\mathit{a}})$ and reconstructed features $D_2({\mathit{I}}_{\mathit{rec}},{\mathit{L}}_{\mathit{a}})$ in the discriminator to be close together, so it can effectively enhance the semantic similarity between ${\mathit{I}}_{\mathit{rec}}$ and the reference image ${\mathit{I}}_{\mathit{a}}$ while ensuring the authenticity of the reconstructed image ${\mathit{I}}_{\mathit{rec}}$.

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{adv}^{D_1}={∥D_1({\mathit{I}}_{\mathit{g}},{\mathit{L}}_{\mathit{g}})-1∥}_1\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{adv}^{D_2}={∥D_2({\mathit{I}}_{\mathit{rec}},{\mathit{L}}_{\mathit{a}})-D_2({\mathit{I}}_{\mathit{rec}},{\mathit{L}}_{\mathit{a}})∥}_1\hfill \end{array}$$

(21)

The total losses of the generation pipeline and the reconstruction pipeline are defined as follows:

$$\begin{array}{cc}\hfill & {\mathcal{L}}_g={\lambda }_{app}\times {\mathcal{L}}_{app}^g+{\lambda }_{adv}\times {\mathcal{L}}_{adv}^{D_1}\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill & {\mathcal{L}}_r={\lambda }_{app}\times {\mathcal{L}}_{app}^r+{\lambda }_{adv}\times {\mathcal{L}}_{adv}^{D_2}+{\lambda }_{pc}\times {\mathcal{L}}_{pc}\hfill \end{array}$$

(23)

where, ${\lambda }_{app}$, ${\lambda }_{adv}$ and ${\lambda }_{pc}$ denote the weighting factors.

3. Results and Discussion

3.1. Implementation Details

Our proposed model is implemented in PyTorch v1.2.0. The model is optimized using Adam optimizer [31] with decay index ${\beta }_1$ = 0, ${\beta }_2$ = 0.9, and learning rate $l_r$ = 0.0001. The weights ${\lambda }_{rec}$, ${\lambda }_{adv}$ and ${\lambda }_{pc}$ of the loss function are set to 1, 0.1 and 0.1, respectively. On a single NVIDIA 2080Ti (11 GB), we trained our model on CelebA-HQ for three days with a batch size of 4. The convergence processes of the loss function and the test set evaluation metrics are shown in Figure 6 and Figure 7. We found that the model achieved the best results at the 38th epoch.

3.2. Completion Consistency

A comparison is made between RG-DP-FICN proposed in this paper and generative image inpainting with contextual attention (CA) [12], generative landmark guided face inpainting (LaFIn) [20], and pluralistic image completion(PIC) [15] on the same database CelebA-HQ. Notice that for CA, LaFIN and PIC, the pre-trained models on the CelebA-HQ are given, these methods are tested utilizing the code and its network weights provided by the original authors.

To visually demonstrate the superiority of the face image completion approach proposed in this paper, a qualitative evaluation is performed for the completion results of this approach, CA, LaFIn, and PIC (Figure 8). The completion result of PIC and CA are essentially average faces based on the entire dataset, due to their identity are uncertain. Therefore, their completion results can possess certain realism, however, the identity is clearly different from the ground-truth. For LaFIn, the landmarks is introduced as prior knowledge of topological structure, nevertheless, its completion result still has no reasonable identity owing to the lack of identity information guidance, particularly manifests as the different contour of eyes, eyebrows, nose from that in the original image. For RG-DP-FICN, the reference image of the same identity is introduced to guide the completion process. Utilizing the identity transfer module for transferring the identity of the reference image to the damaged image, the identity features obtained by decoupling and the posture features of the damaged image are fused. Moreover, the background information lost during identity transfer is restored by the attention feature fusion module. In conclusion, RG-DP-FICN can generate the completed images with reasonable topological structure and identity compared to the above three schemes.

To objectively demonstrate the superiority of the face image completion approach proposed in this paper, the completion results of this approach, CA, LaFIn, and PIC are quantitatively assessed. The quantitative evaluation results are represented in

Table 1. The quantitative results on the CelebA-HQ test set consisting of 1000 face images.

Method	PSNR	SSIM	L1 Loss(%)
CA [12]	24.39	0.863	2.88
PIC [15]	24.86	0.871	2.65
LaFIn [20]	26.22	0.905	2.37
Our method	28.44	0.938	1.58

. As can be seen from the numbers in Table 1, LaFIn is superior over PIC and CA in most cases, as it employs the landmark information to guide completion process. PSNR and SSIM of RG-DP-FICN are far higher than those of other schemes, while L1 loss is lower compared to the other schemes. It is inferred that in comparison with the existing advanced schemes, the RG-DP-FICN proposed in this paper has a stronger capability of face image completion.

3.3. Completion Diversity

To further validate the model’s identity transfer capabilities, the reference face images of different identities are chosen for the damaged images utilizing the center mask (Figure 9 and Figure 10). Covering the face attributes completely, the completion results will represent obvious differences in the position and contour of the eye and the shape of the nose based on the different reference face images. It can be seen that the completion network in this paper effectively guided the identity of the completion result.

4. Conclusions

Considering the problem of identity uncertainty in the results of face image completion, RG-DP-FICN is proposed in this paper. First, an identity transfer module is designed to realize the identity guidance of face image completion results. Then, an attention feature fusion module is designed to effectively restore the background information of the face image. In comparison with various advanced approaches, the completion results of RG-DP-FICN are more real and natural in the subjective visual effect and possessed significantly enhanced objective pixel-level similarity and overall similarity. Hence, it effectively solves the above problem.