An Image Generation Method of Unbalanced Ship Coating Defects Based on IGASEN-EMWGAN

Abstract

During the process of ship coating, various defects will occur due to the improper operation by the workers, environmental changes, etc. The special characteristics of ship coating limit the amount of data and result in the problem of class imbalance, which is not conducive to ensuring the effectiveness of deep learning-based models. Therefore, a novel hybrid intelligent image generation algorithm called the IGASEN-EMWGAN model for ship painting defect images is proposed to tackle the aforementioned limitations in this paper. First, based on a subset of imbalanced ship painting defect image samples obtained by a bootstrap sampling algorithm, a batch of different base discriminators was trained independently with the algorithm parameter and sample perturbation method. Then, an improved genetic algorithm based on the simulated annealing algorithm is used to search for the optimal subset of base discriminators. Further, the IGASEN-EMWGAN model was constructed by fusing the base discriminators in this subset through a weighted integration strategy. Finally, the trained IGASEN-EMWGAN model is used to generate new defect images of the minority classes to obtain a balanced dataset of ship painting defects. The extensive experimental results are conducted on a real unbalanced ship coating defect database and show that, compared with the baselines, the values of the ID and FID scores are significantly improved by 4.92% and decreased by 7.29%, respectively, which prove the superior effectiveness of the proposed model in this paper.

1. Introduction

The shipbuilding industry is an indispensable and important part of the national equipment manufacturing industry, the epitome of modernized industry, and a strategic industry related to national economic development and national defense security. Ship painting refers to the grinding, sandblasting, rust removal, and coating of the hull structure and outfitting parts. As one of the three pillars of the modern shipbuilding process [1], ship painting runs through the whole process of shipbuilding from design and construction to the ship’s delivery [2]. Since each part of the hull is in various corrosive environments, it requires applying different anti-corrosive coatings to its surface before the ship is launched [3]. These coatings not only serve as decorative aesthetics but also function as an effective anti-fouling barrier, which will be critical to the structural integrity, hydrodynamic properties, and longevity of service for the vessel [4]. Affected by the improper operation of the workers, environmental changes during drying and curing, or the quality of the paint itself, diverse defects such as sagging and orange skin are produced during the process of the ship painting [5]. These coating defects are roughly divided into wet film defects and dry film defects. Among them, the wet film defects are generated immediately after the coating is applied to the substrate surface, while the dry film defects are generated during the drying and curing stages of the coating and when the coating is put into service.

The quality of the coating is directly related to the construction cycle and maintenance cost of the ship, and it is an important factor affecting the corrosion resistance of the hull and the service life of the ship [6]. The coating defects detection and filling phase is a key section of the coating PDCA comprehensive quality management. At present, the detection of coating defects in most domestic shipyards involves detecting and judging the coating quality and recording the types and grades of defects by the staff at a specific time after the coating operation is completed, which not only increases the intensity and pressure of human work but also greatly increases the time cost and reduces the efficiency of ship painting operations [7]. Therefore, in the ship coating industry, it is essential to identify the painting defects intelligently and feedback information such as the type, size, shape, and location of each defect in the coating process. With the proposal of ship intelligent manufacturing theory and the continuous popularization of artificial intelligence technology in the ship painting industry, people began to gradually apply the intelligent technology to the knowledge acquisition of ship painting defects, but there are fewer reports on the application of image-based ship coating defects recognition.

Machine learning (ML) is one of the core research topics of artificial intelligence and neural computing [8], which is extensively applied to solve the problems of classification and prediction. Generalization ability has always been one of the important criteria for evaluating the models of ML algorithms. However, due to various reasons such as a high tagging cost, data security, and privacy protection, there is no dedicated and large dataset of ship painting defects images for the training of deep learning models, which are highly susceptible to small sample problems such as model overfitting, low classification accuracy, and falling into local optimum solutions. In particular, the characteristics of class imbalance in the ship coating defect dataset [9] lead to the fact that the traditionally trained classification model cannot identify the minority classes easily, thus becoming an overfitting model and making its generalization performance greatly reduced.

Generative models-based neural networks can alleviate the above problems [10,11]. Generative Adversarial Networks (GAN) and its variants [12] are becoming one of the most popular generative network models due to their low loss and end-to-end advantages in generating images, and they have recently been successfully applied to image generation [13], image-to-image translation [14], image super-resolution enhancement [15], anomaly detection [16], and other applications. However, there are still problems such as difficulty in generating controllable multi-class and high-quality datasets [17], unstable training [18], and pattern collapse [19] because GANs are unsupervised learning. Therefore, many recent efforts on GANs have focused on overcoming these inherent drawbacks by developing various adversarial training approaches such as optimizing the objective loss function, training additional discriminators, training multiple generators, and using the neural-evolution computation strategy (NECS) [20]. One of them is EGAN, proposed by Wang et al. [21], which is most widely used in image generation due to its combination of methods such as the neural-evolution computation strategy (NECS) and adversarial training mechanism. XIAO et al. [22] proposed a rumor propagation model based on GAN and anti-rumor for enhancing homomorphism data in the sample space, and the experiments showed that the proposed method achieves 9%–29% over the current methods in terms of precision gains. Chen et al. [23] proposed a hybrid algorithm combining the Generative adversarial nets (GAN) and Genetic Algorithm (GA), and the experimental results showed that the proposed method can solve the permutation flow shop problem to verify the algorithm’s solution ability. Han et al. [24] proposed a Generative Adversarial Network with Evolutionary Generators (EG-GAN) and applied it in face image inpainting. Experiments on various face image datasets show that EG-GAN successfully overcomes the gradient vanishing problem, achieves stable and efficient training, and generates visually reasonable images. Erivaldo et al. [25] proposed an algorithm for pruning GANs based on Evolution Strategy (ES) and Multi-Criteria Decision Making (MCDM) and applied it in medical imaging diagnosis. The extensive experimental results show that the pruned GAN model decreases to 70% Floating-Point Operations when compared with the original model. Meng et al. [26] proposed a novel hybrid short-power prediction model in newly built wind farms based on secondary evolutionary generative adversarial networks (SEGAN) and a dual-dimension attention mechanism (DDAM)-assisted bidirectional gate recurrent unit (BiGRU) for solving the insufficient data. The experiments were conducted on Galicia Wind Farm in Sotavento, and the results show that the proposed model yields successfully. Zheng et al. [27] proposed a novel evolutionary generative adversarial network (GAN) ensemble method for HSR passenger classification. Compared with state-of-the-art methods, the proposed model has exhibited significant advantages and has successfully been applied to the anti-terrorism for the Chinese HSR. He et al. [28] developed a new evolvable adversarial framework for automatic COVID-19 infection segmentation. Experiments on several COVID-19 CT scan datasets verified that the proposed method achieved superior effectiveness and stability for COVID-19 infection segmentation. Cai et al. [29] proposed a skin cancer detection model based on federated learning integrated with a deep generation model based on the knee point-driven evolutionary algorithm (KnEA). Experiments were conducted on the ISIC 2018 dataset, and the results show that the proposed model can help resolve problems of insufficient data in smart medicine of IoT. Hazra et al. [30] proposed a generative adversarial networks (GAN) model to create synthetic nucleic acid sequences of the cat genome tuned to exhibit specific desired properties, and the experimental results shows that our proposed architecture could create synthetic nucleic acid sequences efficiently and exhibits desired properties of the cat genome. Talasa et al. [31] proposed a novel method for exploiting Generative Adversarial Networks to simulate an evolutionary arms race between the camouflage of a synthetic prey and its predator. Chen et al. [32] proposed a novel data-driven scenario generation using generative adversarial networks, which is based on two interconnected deep neural networks for stochastic power generation processes. The experiments were conducted on wind and solar time-series data from NREL integration datasets, and the results show that the proposed method is able to generate realistic wind and photovoltaic power profiles with a full diversity of behaviors.

Although the above research findings have all obtained good optimization results, there is still much room for improvement. Such weaknesses include standard EGAN, and its variants use only a single discriminator to discriminate, leading to large accuracy errors, which cannot guarantee the quality and diversity of the generated samples, leading to pattern collapse and poor training stability. Generators with lower performance are directly discarded and easily fall into local optimal solutions, there is no cross-variation operation to guarantee search performance, and there are other challenges. Therefore, the area is still in an infantile stage, and we believe that the model and work proposed in this paper will be able to make a valid contribution to filling the current research gap of knowledge.

In light of the aforementioned challenges, a novel hybrid defect image generation model based on selective ensemble learning (SEN) and Auxiliary Classifier Wasserstein evolutionary GAN(EGAN)-assisted GA improved by simulated annealing (IGA) is first proposed in this paper to generate new high-quality defect images of the minority classes in ship coating defects, which alleviates the problems of insufficient data and class imbalance to some extent.

The main novelties and contributions of this paper are summarized as follows.

(1)

To address the dilemma of a ship coating defect database’s insufficient historical labeled data and class imbalance, this paper proposes a novel hybrid defect image generation model called IGASEN-EMWGAN for generating the new high-quality defect images of the minority class for the first time.

(2)

In order to alleviate the problem of a vanishing gradient and model collapse in original minimax mutation and provide smoother gradients for updating the generators and stabilizing the training process, we also employ the modified hinge mutation and the wasserstein_gp mutation to inject genome mutation diversity into the training of complementary generators, which is conducive to steady training between generators and discriminators.

(3)

To remedy the deficiencies existing in vanilla EGANs, such as a lack of a crossover mutation operator, it being limited by the single specified environment (discriminator), and it easily getting stuck in the local optimum, we also propose adding the two-point crossover mutations operator (TP) into the proposed IGASEN-EMWGAN model to evolve the generators, which is conducive to fostering diversity in generators and discriminators.

(4)

In order to solve the limitations of those preveniently proposed methods, such as the difficult balance between the discriminative accuracy and diversity of base discriminators and the high possibility of falling in local optima, we propose a novel model, which integrates selective ensemble learning and GA-based SA into the training of multi-discriminators in IGASEN-EMWGAN to escape from the local optimum caused by the class imbalance.

(5)

The extensive experiments are conducted on a real unbalanced ship coating defect database, and the experimental results demonstrate that, compared with the baselines and different state-of-the-art GANs, the values of the ID and FID scores are significantly improved and deceased, respectively, which proves the superior effectiveness of the proposed model in this paper.

The rest of the paper is organized as follows: the details of the proposed IGASEN-EMWGAN model are illustrated in Section 2. Section 3 describes the experimental design and gives the experimental results. The conclusions of the paper and the outlook for future research are presented in Section 4.

2. Proposed Method

In this section, we first review the original E-GAN structure. Then, the algorithm framework of the IGASEN-EMWGAN model is proposed to remedy the above challenges. The implementation details of the proposed IGASEN-EMWGAN model are discussed by explaining the whole modified neuro-evolutionary process of base generators and discriminators in detail. Among them, the whole evolutionary process of base generators in IGASEN-EMWGAN includes four steps: variation mutations, crossover mutations, evaluation, and selection, while the base discriminators in IGASEN-EMWGAN contain five steps: generation, encoding, evolution (variation mutations, crossover mutations, evaluation, and selection based on ensemble pruning), update, and combination.

2.1. Evolutionary Generative Adversarial Networks

The proposed classification algorithm for ship coating defects consists of four modules, namely, a data acquisition module, a data pre-processing module, a data generation module, and a coating defect classification module, as shown in Figure 1.

Evolutionary Generative Adversarial Networks (E-GAN) were first proposed in [22] to mitigate the inherent deficiencies existing in conventional GANs, namely, mode collapse, training instability, and vanishing gradients. Different from vanilla GANs, which update a generator and a discriminator iteratively in a specified adversarial optimization strategy, E-GAN introduced a neuro-evolutionary computation strategy (NECS) that evolved a population of generators (parents) {$G_{\theta }$} instead of a single generator in a given static environment denoted by the discriminator $D_{\varphi }$ to produce a set of new generators (offspring). The adversarial training procedure of the generator and discriminator is treated as an evolutionary process of individual species in response to the environment. During the process of evolution, each evolution mainly includes the following three steps: variation, evaluation, and selection, as shown in Figure 1a.

2.2. Generators in IGASEN-EMWGAN

In summary, most existing generators of E-GANs lack a crossover mutation operator, are limited by the single specified environment (discriminator), and easily get stuck in the local optimum, which has a big influence on the optimization performance during training. To this end, apart from the two original genetic operators (variation mutations operator and selection operator) in vanilla E-GAN, we also utilize the crossover mutations operator to evolve the generators in IGASEN-EMWGAN. In addition to it, we build a set of generators $\left\{G_{{\theta }^j}\right\}(j=1,2,\cdots ,\mu )$ (acting as parents) to produce a population of new generators (acting as offspring) $\left\{G_{{\theta }^{j,c}}\right\}(j=1,2,\cdots ,\mu ;c=1,2,\cdots ,M)$ to adapt to the dynamic environment (i.e., discriminators $\left\{D_{{\varphi }^n}\right\}(n=1,2,N)$), which represents that the evolved generators can generate controllable and higher-quality images and estimate the distribution of true data accurately.

Based on the modified NECS, we take Variation mutations, Crossover mutations, Evaluation, and Selection for evolving the generators in IGASEN-EMWGAN training. The whole modified neuro-evolutionary computation process of generators in IGASEN-EMWGAN is depicted in Figure 1b and Algorithm 1.

2.2.1. Variation Mutations of Generators

In essence, the variation mutations of generators in IGASEN-EMWGAN are the different objective functions, whose purposes are to update the parameters in generators followed by each evaluation by the discriminators and narrow the distances between the generated distribution and the data distribution [22,33]. As shown in Figure 2, the performance of the original minimax mutation is disappointing because the gradient of the generator easily vanishes. To this end, we employ the modified hinge loss as the hinge mutation in this section, which replaces the original minimax mutation and expects the final discriminator to discriminate the generated samples from real samples as true as possible (i.e., $D_{\varphi }(G_{\theta }(z),C)\to 1$) or to be punished in IGASEN-EMWGAN. The modified hinge mutation introduces class label C, and the term of gradient penalty (GP) to the original hinge loss, which assists generators in generating specific types of samples and achieves Lipschitz continuity constraints, respectively. The modified hinge mutation can be defined as follows:

$$M_G^{hinge}={\displaystyle \sum _{n=1}^{\tau }\left\{E_{z\sim p_z}[\max (0,1-{\tilde{D}}_{{\varphi }^n}(G_{\theta }(z),C)]+E_{x\sim p_{data}}[\max (0,1+{\tilde{D}}_{{\varphi }^n}(x))]\right\}+\xi E_{\widehat{x}\sim p_{\widehat{x}}}}[{({‖{\nabla }_{\widehat{x}}D(\widehat{x})‖}_2-1)}^2]$$

(1)

where the $\tilde{D}(x)$ and $\tilde{D}(G_{\theta }(z),C)$ in this paper are defined as follows:

$${\tilde{D}}_{{\varphi }^n}(x)=D_{{\varphi }^n}(x)-E_{x\sim p_{data}}{D}_{{\varphi }^n}(x)$$

(2)

$${\tilde{D}}_{{\varphi }^n}(G_{\theta }(z),C)=D_{{\varphi }^n}(G_{\theta }(z),C)-E_{z\sim p_z}{D}_{{\varphi }^n}(G_{\theta }(z),C)$$

(3)

where the noise sample $z\sim p_z$ is from a normal or uniform distribution, the real sample $x\sim p_{data}$ is the distribution of real samples x, C denotes the class labels of ship coating defect images, $(G_{\theta }(z),C)$ is the generated image with class label C, and $D_{{\varphi }^n}(G_{\theta }(z),C)$ is the output of the discriminators.

Apart from the hinge mutation, we also employ a novel wasserstein_gp mutation inspired by WGAN-GP [34,35] and ACGAN [36] to provide smoother gradients for updating the generators and stabilizing the training process, as shown in Figure 2. Unlike the mutations in original EGANs, the wasserstein_gp mutation proposed in this paper not only generates value functions superior to those generated through the Jensen–Shannon divergence (JSD), which alleviates the vanishing gradient and mode collapse, but also replaces weight clipping with a gradient penalty to satisfy the Lipschitz continuity and constraint, which is beneficial to improving the speed and stability of training [37]. In a nutshell, using the combination of hinge mutation and wasserstein_gp mutation as mutation operators of generators in IGASEN-EMWGAN will not only avoid these deficiencies such as gradient vanishing and mode collapse to some extent but will also inject genome mutation diversity into the training of complementary generators, which is conducive to steady training between generators and discriminators. The wasserstein_gp mutation is defined in Equation (4).

$${{\rm M}}_G^{wassestein\_gp}={\displaystyle \sum _{n=1}^{\tau }{{\rm E}}_{z\sim p_z}[D_{{\varphi }^n}(G_{\theta }(z),C)]}+E_{z\sim p_z}[P(class=c|G_{\theta }(z))]$$

(4)

2.2.2. Crossover Mutations of Generators

Another idea in modified NECS for addressing the training pathologies of conventional GANs is also motivated by the crossover mutation operators in genetic operators. As an equally important mutation operator in evolutionary algorithms (EAs) [38], the operators of crossover mutation can exchange the genes in chromosomes between two paired parents to help offspring inherit some superior traits from the parents, which is conducive to improving the diversity of the genome and the global search ability. Common crossover operators include single-point crossover (SP), two-point crossover (TP), scattered crossover (SC), heuristic crossover (HE), arithmetic crossover (AM), and intermediate crossover (IT) [39,40]. In this paper, we propose adding a two-point crossover operator (TP) into the IGASEN-EMWGAN model to foster diversity in generators, which will therefore be better than existing EGANs.

2.2.3. Evaluation of Generators

In the modified neuro-evolutionary computation strategy, evaluation is the operator of measuring the quality and diversity of evolved individual offspring (i.e., children) by the changing environment (i.e., discriminators $D_{{\varphi }^n}$). In this paper, we devise an evaluation function which consists of two typical properties (i.e., quality fitness score $F_{G^{quality}}$ and diversity fitness score $F_{G^{diversity}}$) of the generated samples to evaluate the performance of the evolved offspring of generators for determining the evolution direction (i.e., individual offspring selection) in IGASEN-EMWGAN. The formal representations of the quality fitness score ($F_{G_{\theta }^{quality}}$) and diversity fitness score ($F_{G_{\theta }^{diversity}}$) are as follows:

$$F_{G_{\theta }^{quality}}={\displaystyle \sum _{n=1}^{\tau }{E}_{z\sim p_z}[D_{{\varphi }^n}(G_{\theta }(z),C)]}$$

(5)

$$F_{G_{\theta }^{diversity}}={\displaystyle \sum _{n=1}^{\tau }\log ||{\nabla }_{D_{{\varphi }^n}}-E_{x\sim p_{data}}[\log D_{{\varphi }^n}(x)]}-E_{z\sim p_z}[\log (1-D_{{\varphi }^n}(G_{\theta }(z),C))]||$$

(6)

Overall, based on the aforementioned two function fitness scores, we can finally give the evaluation function of the generators proposed in this paper, which are defined as:

$$F_{G_{\theta }}=F_{G_{\theta }^{quality}}+\lambda F_{G_{\theta }^{diversity}}$$

(7)

where λ ($\lambda \ge 0$) in front of $F_{G_{\theta }^{diversity}}$ is used to balance the two indices of the generative quality fitness score $F_{G_{\theta }^{quality}}$ and diversity fitness score $F_{G_{\theta }^{diversity}}$ of the generated samples. In short, a relatively high fitness score $F_{G_{\theta }}$ leads toa higher training efficiency and better generative performance.

2.2.4. Selection of Generators

The selection operator, which acts as the counterpart of the mutation operators and recombination, aims to select individuals (parents) to produce the next well-performing offspring population. The most common methods are roulette wheel selection [41] and binary tournament selection [42]. In the proposed IGASEN-EMWGAN, we employ the simple yet used comma survivor selection strategy, i.e., (μ, λ)-selection [43], as the selection mechanism of IGASEN-EMWGAN to select and retain the desired ones based on the fitness score of existing individuals of generators. We define the selection fitness function score for the offspring populations of generators in IGASEN-EMWGAN as:

$$\left\{F_{G_{\theta }}^{1,1},F_{G_{\theta }}^{2,1},\cdots ,F_{G_{\theta }}^{\mu ,M}\right\}\leftarrow sor{t}_{\max }(\left\{F_{G_{\theta }}^{j,\nu }\right\})$$

(8)

After sorting the current μ offspring populations (generators) $\left\{G_{{\theta }^1},G_{{\theta }^2},\cdots ,G_{{\theta }^{\mu }}\right\}$ in ascending order by fitness function scores, the μ best-performing individuals that survived are selected to form the generation of the next evolution during evolutionary adversarial training in IGASEN-EMWGAN. Therefore, it can be written as:

$${\theta }^1,{\theta }^2,\cdots ,{\theta }^{\mu }\leftarrow {\theta }^{1,1},{\theta }^{2,1},\cdots ,{\theta }^{\mu ,1}$$

(9)

2.3. Discriminators in IGASEN-EMWGAN

In fact, although many recent attempts regarding GANs have been made by building multi-discriminators and have contributed to overcoming many well-documented failures (i.e., lack of informative feedback gradient, larger discrimination deviation of a single discriminator) existing in GAN models to some extent [44,45,46], GANs can easily stop adversarial training and get stuck in a local optimal solution when they try to learn the distributions from the minority classes in unbalanced datasets [47,48,49], as shown in Figure 3a. Apart from it, the tradeoff metric between the discrimination accuracy and the difference in multiple base discriminators has no well-accepted, unified, formal definition [50], resulting in decreasing the generalization performance and increasing the hardware memory and computational costs of the server system. To this end, considering the aforementioned limitations of those preveniently proposed methods, we propose a novel model, which integrates selective ensemble learning and a genetic algorithm-based simulated annealing algorithm into the training of multi-discriminators in IGASEN-EMWGAN to escape from the local optimum caused by class imbalance and improve the training efficiency and the generalization performance. The overall modified NECS process of multi-discriminators in IGASEN-EMWGAN is presented in Figure 1b and illustrated in Algorithm 1.

2.3.1. Generation of Base Discriminators

Before the evolutionary process of the base discriminators in IGASEN-EMWGAN, the random construction of different base discriminators is crucial for improving the discrimination accuracy and the generalization ability of the ensemble model. In this paper, to warranty the variety of the constructed base types, we first use the Bootstrap sampling algorithm [51] to construct τ multiple training subsets $su{b}_i(i=1,2,\cdots ,\tau )$ with a strong diversity degree from $S_{train}$. Subsequently, it is necessary to initialize the weights for the minority class and majority class in a subset of training samples, which can reduce the effect of the initial imbalance data, make the initial base discriminator have better discriminative ability, and ensure the diversity of the base discriminator. The weights of minority class sample weights and majority class samples are shown in Equations (10) and (11), respectively. A combination of the training data sample perturbation and algorithm parameter perturbation methods is proposed to train initial homogenous base discriminators with diversities independently [52]. Among them, the k-fold cross-validation (CV) method [53], for its simplicity and universality, is regarded as an effective method for perturbing data samples, where the training set is divided into k equal parts (one for testing and the remaining k-1 folds for training). In this paper, we choose the five-fold cross-validation method, which gives robustness against overfitting and underfitting, as shown in Figure 4. Apart from the training data sample perturbation method, it is crucial for the proper tuning of the hyperparameters, which ensures the goodness-of-fit of the outcomes and the model fitting convergence. The common algorithms include Grid Search, Random Search, and Bayesian Optimization (BO) [54]. In this paper, the optimization strategy of Grid Search is used to set different network hyperparameters (i.e., number of hidden layer neurons, number of layers, and initial connection weights) to construct the subsets of candidate base discriminators with large diversities so as to obtain different compact sets of candidate initial base discriminators randomly.

$$w_{-}=\frac{\left|S_{+}\right|}{\left|S_{-}\right|\left|S\right|}$$

(10)

$$w_{+}=\frac{\left|S_{-}\right|}{\left|S_{+}\right|\left|S\right|}$$

(11)

where $\left|S\right|$ denotes the total number of samples in the training set, and $\left|S_{-}\right|$ and $\left|S_{+}\right|$ represent the number of samples in the minority class and the majority class in the training set, respectively.

2.3.2. Encoding of Base Discriminators

For evolutionary algorithms such as GA, the first step is to connect a bridge in the context of a real situation problem and the solving problem space through the NECS [55]. In order to represent the candidate base discriminators in the search space, a desired encoding scheme is adopted, where each of the chromosomes is represented by the vectors’ length. The common coding methods are binary coding, gray coding, floating-point (real) coding, permutation coding, and so forth [56]. Consequently, for the selected optimal base discriminator subset, all the homogeneous candidate base discriminators $D_{\varphi }{}^i(i=1,2,\cdots ,\tau )$ are encoded as the binary chromosome individuals in the genetic space, where 1 means the base discriminators are selected for the ensemble member, while 0 means the opposite, as shown in Figure 5 [57].

2.3.3. Evolution of Base Discriminators

To provide informative gradients to the generator through the adversarial training, we employ the modified neuro-evolutionary computation strategy (NECS) to generate multiple offspring discriminators. The specific flowchart of updating the discriminators in IGASEN-EMWGAN is shown in Figure 6. The whole modified neuro-evolutionary computation strategy (NECS) includes four steps: variation mutations, crossover mutations, evaluation, and selection.

➀

Variation mutations of discriminators

Similar to the variation mutations of generators in IGASEN-EMWGAN, various objective functions are taken as variation mutations for base discriminators. In this work, we employ two complementary variation mutation operators, including minimax mutation and least-squares mutation to provide a promising optimization direction for the evolution of the base discriminators in IGASEN-EMWGAN. The operators of the minimax mutation and least-squares mutation are formulated as:

$$M_D^{\mathrm{mini}\max }=E_{x\sim p_{data}}[\log D_{{\varphi }^n}(x)]+E_{z\sim p_z}[\log (1-D_{{\varphi }^n}(G_{\theta }(z),C))]$$

(12)

$$M_D^{least-squares}=\frac{1}{2}{E}_{x\sim p_{data}}[{(D_{{\varphi }^n}(x)-1)}^2]+\frac{1}{2}{E}_{z\sim p_z}[D_{{\varphi }^n}{(G_{\theta }(z),C)}^2]$$

(13)

➁

Crossover mutations of base discriminators

As one of the important steps in GAs for maintaining dominant sequences and increasing sample diversity [32], the operators of crossover mutations are another key factor for fostering the diversity of base discriminators. Similar to the operators of the crossover mutations, we also add a two-point crossover operator (TP) into the IGASEN-EMWGAN model to update the parameters in base discriminators, which will therefore be better than the existing EGANs in this paper.

➂

Evaluation of base discriminators

Generally, the appropriate handling of the tradeoff between diversity and the performance of the ensemble discriminators is crucial to the effectiveness of ensemble learning [58,59]. From the perspective of the measurement of diversity, there are a substantial number of statistics that can measure the diversity among the sub-discriminators, such as the Q statistics, the correlation, and the entropy of the votes, which can be grouped into two categories: pairwise diversity measures and non-pairwise diversity [60]. The interrater agreement [61,62], which is considered to measure the intraclass correlation coefficient, is employed. Because of the influence of the unbalanced dataset, a modified interrater agreement ${\kappa }^{\prime }$ is proposed in this paper and defined as follows:

$${\kappa }^{\prime }=1-\frac{1}{2\overline{p}(1-\overline{p})}Di{s}_{av}$$

(14)

where $\overline{p}$ denotes the average discrimination accuracy of K base discriminators for the training set $S_{train}=\left\{\left({\overrightarrow{x}}_1,l_1\right),\left({\overrightarrow{x}}_2,l_2\right),\cdots ,\left({\overrightarrow{x}}_s,l_s\right)\right\}$ and is expressed as

$$\overline{p}=\frac{1}{M{K}^2}{\displaystyle \sum _{s=1}^{MK}{\displaystyle \sum _{i=1}^K{h}_i({\overrightarrow{x}}_s,l_s)}}$$

(15)

where $h_i({\overrightarrow{x}}_s,l_s)$ is described as a correct/incorrect decision, ${\overrightarrow{x}}_s$ and $l_s\in \left\{0,1\right\}$ represent the samples with n-dimensional features in the training set, 0 represents the generated samples, and 1 represents the real samples.

The average disagreement measure $Di{s}_{av}$ is defined as follows:

$$Di{s}_{av}=\frac{1}{K(K-1)}{\displaystyle \sum _{i=1}^K{\displaystyle \sum _{\begin{array}{l}j=1\\ i\ne j\end{array}}^{K}Di{s}_{i,j}}}$$

(16)

where $Di{s}_{i,j}$ is defined as follows:

$$Di{s}_{i,j}=\frac{w_1(N_{ij}{}^{01}+N_{ij}{}^{10})+w_0(N_{ij}{}^{01}+N_{ij}{}^{10})}{w_1{N}_{ij}{}^1+w_0{N}_{ij}{}^0}$$

(17)

where $w_0$ and $w_1$ represent the sample weights of the minority class and the majority class, respectively, $N_{ij}{}^{xy}$ denotes the number of samples in the validation set for which the base discriminator $h_i$ outputs x while the base discriminator $h_j$ outputs y for some validation set instance, as shown in

Table 1. The definition of the $N_{ij}{}^{xy}$.

Discriminator (i)	Discriminator (j)
	Predict (0)	Predict (1)
Predict (0)	$N_{ij}{}^{00}$	$N_{ij}{}^{10}$
Predict (1)	$N_{ij}{}^{01}$	$N_{ij}{}^{11}$

, and $N_{ij}{}^0$ and $N_{ij}{}^1$ represent the total number of the generated samples and the real samples, respectively.

Apart from the diversity, the discrimination accuracy of the basic discriminators is another important complementarity factor for improving the ensemble performance. In practical applications, the metrics of the F1-score [63] and G-mean [64] are more effective for evaluating the model for unbalanced data. Therefore, to evaluate the discrimination performance of the subpopulation evolved by the discriminators, we devise an evaluation characteristic called Accuracy–Diversity (AD), which considers both the discrimination accuracy (a combination of F1-score and G-mean) and the modified interrater agreement ${\kappa }^{\prime }$ of the ensemble model in building the selection criterion simultaneously. The metric of AD can be formulated as

$$AD=\beta \times Value(F1-score)+\gamma \times Value(G-mean)+\left(1-\beta -\gamma \right)\times {\kappa }^{\prime },\beta , \gamma \in [0,1]$$

(18)

where β and γ are the weight coefficients balancing between the F1-score, G-mean, and modified interrater agreement ${\mathbf{\kappa }}^{\mathbf{\prime }}$ of the models of basic discriminators.

➃

Selection of base discriminators

The evaluation procedure corresponds to a selection process, which selects the best-performing evolved individuals with larger evaluation scores [65]. After evaluation, similar to the selection of the generators in IGASEN-EMWGAN, the new parents of the next evolution of base discriminators in IGASEN-EMWGAN will be selected with respect to the value of AD. The selection function for the offspring of the base discriminators is defined as

$$\left\{F_{D_{\varphi }}^{1,1},F_{D_{\varphi }}^{2,1},\cdots ,F_{D_{\varphi }}^{\tau ,N}\right\}\leftarrow sor{t}_{\max }(\left\{F_{D_{\varphi }}^{\tau ,c}\right\})$$

(19)

After sorting the current τ offspring populations (base discriminators) $\left\{D_{{\varphi }^1},D_{{\varphi }^2},\cdots ,D_{{\varphi }^{\tau }}\right\}$ by descending order according to the value of AD, the τ best-performing component individuals that survived are selected to form the generation of the next evolution during the evolutionary adversarial training in IGASEN-EMWGAN. Therefore, it is formulated as follows:

$${\varphi }^1,{\varphi }^2,\cdots ,{\varphi }^{\tau }\leftarrow {\theta }^{1,1},{\theta }^{2,1},\cdots ,{\theta }^{\tau ,1}$$

(20)

2.3.4. Update of Discriminators

To avoid the premature convergence and increase the possibility of finding new solutions in the population rather than fall into the local optima, this paper introduces the mechanism of simulated annealing (SA) [66] into the training process of updating the discriminators to select after retaining the best-fit offspring of base discriminators. In addition to the best-performing discriminators with high discriminatory accuracy and variability by the metric of AD, a subset of base discriminators with poor discrimination performance will be likely to be retained according to the Metropolis criterion, which is conducive to avoiding the local optimum trapping problem. The Metropolis criterion is defined as follows:

$$p=\left\{\begin{array}{cc}1,& E(n+1)\prec E(n)\\ \exp (-\frac{E(n+1)-E(n)}{T}),& E(n+1)\ge E(n)\end{array}\right.$$

(21)

where p represents the probability of accepting a new base discriminator, E(n + 1) and E(n) denote the AD value of the base discriminator with modified network parameters and the AD value of the previous base discriminator, respectively.

2.3.5. Combination of Discriminators

To obtain the best and most robust discriminator model, it is necessary to form a combined discriminator model. In this study, the chromosomes corresponding to the shortlisted evolved best-performing discriminators by the modified NECS are at first decoded. Then, the shortlisted evolved best-performing base discriminators and the suboptimal discriminator by the mechanism of SA are combined by the weighted ensemble strategy. In this paper, the weighted ensemble strategy is based on the values of AD. Suppose the weight of the best-performing discriminators by the modified NECS offspring discriminator is $w_1$ and the other weight of the suboptimal base discriminator is $w_2$. The relationship between $w_1$ and $w_2$ is shown in Equations (22) and (23).

$$\frac{w_1}{w_2}=\frac{AD(D_{\varphi -best}^{i,c})}{AD(D_{\varphi -suboptimal}^{i,c})}$$

(22)

$$w_1+w_2=1$$

(23)

Algorithm 1: IGASEN-EMWGAN

Require: mini-batch size ψ; the number of iterations T; the generator $G_{\theta }$; the discriminator $D_{\varphi }$; the updating steps of the discriminator per iteration $n_D$; the number of parents for generator $\mu$; the number of parents for the base discriminator $\tau$; the number of mutations for generator M; the number of mutations for discriminator N; the number of variation mutations $n_m$; the number of crossover mutations $n_c$; the spatial dimension $d_z$ of the noise z; the initial temperature of annealing $T_0$; the annealing coefficient α; the hyperparameter $\gamma$ of the fitness function of generators; the hyperparameter δ of the fitness function of base discriminators. Initialize base discriminators, parameters $\left\{{\varphi }_0^1,{\varphi }_0^2,\cdots ,{\varphi }_0^{\tau }\right\}$ and generators, parameters$\left\{{\theta }_0^1,{\theta }_0^2,\cdots ,{\theta }_0^{\mu }\right\}$. 1: Construct $\tau$ multiple training subsets $su{b}_i(i=1,2,\cdots ,\tau )$ with a diversity degree from $S_{train}$ by using the Bootstrap Sampling Algorithm. 2: Obtain the initial homogeneous base discriminators set $D_{\varphi }{}^i(i=1,2,\cdots ,\tau )$ by using these multiply training subsets $su{b}_i(i=1,2,\cdots ,\tau )$ to train independently, and each parameter is optimized by the parallel optimization strategy. ▷Discriminators Generation 3: Encoding the homogeneous candidate base discriminators $D_{\varphi }{}^i(i=1,2,\cdots ,\tau )$ as the binary chromosome individuals in the genetic space, where 1 means the base discriminators are selected for an ensemble member, while 0 means the opposite. ▷Discriminators Encoding 4: for t =1, …, T do 5: for i =1, …, $n_D$ do 6:   for n = 1, …, τ do ▷Discriminators Evolution 7:    ${\left\{x^{(s)}\right\}}_{s=1}^{\psi }\sim p_{data}$←mini-batch sampling randomly from the real ship coating defects training set. 8:   ${\left\{z^{(s)}\right\}}_{s=1}^{\psi }\sim {\mathrm{P}}_z$←mini-batch sampling randomly from noise samples, and generate a batch of generated samples. 9:    $D_{\varphi }{}^i(i=1,2,\cdots ,\tau )$ generates N offspring $D_{\varphi }^{i,n}(n=1,2,\cdots ,N)$ via Equation (12) or Equation (13), respectively.  ▷D-Variation Mutation 10:   $D_{\varphi }^{i,n}(n=1,2,\cdots ,N)$ generates N offspring $D_{\varphi }^{i,c}(c=1,2,\cdots ,N)$ via D-crossover mutation, that is, updating $D_{\varphi }^{i,n}(n=1,2,\cdots ,N)$. ▷D-Crossover Mutation 11:    Calculate the individual fitness $\left\{F_n\right\}(n=1,2,\cdots ,N)$ of the N evolved offspring of discriminators via Equation (16). ▷D-Evaluation 12:     Sort $\left\{F_n\right\}(n=1,2,\cdots ,N)$, and express the largest one as $F_{nbest}$. ▷D-Selection 13:   end for 14:   if $F_{nbest}$ > $F_D$ then ▷D-Update 15:      Update $D_{nbest}$ to $D_{new}$ 16:   else 17:      $\Delta =F_{cbest}-F_D$ 18:      $P=e^{-\Delta /T}$ 19:      Update $D_{nbest}$ to $D_{new}$ with a probability of P which ranges from 0 to 1 20:   end if 21:      $T_i=\alpha \ast T_{i-1}$ 21:    end for 22:  end for 23:  Output the remaining filtered base discriminators combination. ▷D-Combination 24:  for j = 1,…, $\mu$ do ▷Generators Evolution 25:   ${\left\{z^{(s)}\right\}}_{s=1}^{\psi }\sim {\mathrm{P}}_z$←mini-batch sampling randomly from noise samples. 26:  $G_{\theta }{}^j(j=1,2,\cdots ,\mu )$ generates M offspring $G_{\theta }^{j,m}(m=1,2,\cdots ,M)$ via variation mutation, that is, updating $G_{\theta }{}^j(j=1,2,\cdots ,\mu )$ via Equations (1) and (4), respectively. ▷G-Variation Mutation 27:  $G_{\theta }^{j,m}(m=1,2,\cdots ,M)$ generates M offspring $G_{\theta }^{j,\upsilon }(\upsilon =1,2,\cdots ,M)$ via G-variation mutation, that is, updating $G_{\theta }^{j,m}(m=1,2,\cdots ,M)$. ▷G-Crossover Mutation 28:   Calculate the individual fitness $\left\{F_m\right\}(m=1,2,\cdots ,M)$ of the M evolved offspring of discriminators via Equation (7). ▷G-Evaluation 29:   Sort $\left\{F_m\right\}(m=1,2,\cdots ,M)$, and select the largest offspring $F_{mbest}$ as the next generation’s parents of the generators via Equations (8) and (9). ▷G-Selection 30:   end for 31:end for 32:   Print (New Structure) 33: end

3. Experimental Results and Analysis

In order to qualitatively and quantitatively demonstrate the effectiveness and robustness of our proposed algorithm, namely, IGASEN-EMWGAN, the comparative experiments based on different evaluation metrics are conducted for the sake of demonstrating the merits of the proposed model over the baselines in this subsequent section. The configure information for implementing the experiments involved is 11th Gen Intel (R) Core (TM) i5-11400H @ 2.70GHz with NVIDIA GeForce RTX 3050 Laptop GPU and 16.0 GB RAM. The programming language is Python, and the version is 3.7.4.

3.1. Dataset Setup

In our work, 10 typical types of ship coating defects are analyzed, including Holiday coating, Sagging, Orange skin, Cracking, Exudation, Wrinkling, Bitty appearance, Blistering, Pinholing, and Delamination. The details of the unbalanced ship coating defects dataset are listed in

Table 2. The description of the unbalanced ship coating defects dataset.

Category of Defects	Sample Image	Sample Number (N)	IR
Holiday coating		5	107.2
Sagging		86	6.23
Orange skin		536	1
Cracking		77	6.96
Exudation		71	7.54
Wrinkling		63	8.51
Bitty appearance		13	41.23
Blistering		108	4.96
Pinholing		26	20.62
Delamination		8	67

. Since the size of the collected original ship coating defects images varied in proportion, each image was resized to 128 × 128 × 3 before the experiments were performed. The pre-processed ship painting defect dataset was randomly split into a training set, test set, and validation set in the ratio of 0.64:0.16:0.2. Apart from the aforementioned preprocessing techniques, Both the techniques of data smoothing (gaussian filtering) and data normalization (batch normalization) are used to remove image noise and improve the performance of neural networks significantly, respectively.

3.2. Evaluation Metrics

To evaluate the overall generative performance of the proposed model accurately and comprehensively, different quantitative evaluation indices are adopted to measure the generative performance in our experiments. In this paper, the evaluation metrics we choose are the Inception Score (IS) [67] and the Fréchet Inception Distance score (FID) [68], which are the two most widely used metrics in the training of GANs. Among them, the IS computes the Kullback–Leibler Divergence (KLD) between the conditional class distribution $p(C|x)$ and the marginal class distribution [22], while the FID score calculates the Wasserstein-2 distance between the generated samples and the real samples in the feature space. Therefore, the higher the IS and the lower the FID score, the better the quality and diversity of the generated images. They are formulated as follows:

$$IS(G)=\exp (E_{x\sim p_g}{D}_{KL}(p(C|x)||p(C)))$$

(24)

$$FID=||{\mu }_r-{\mu }_g|{|}^2+Tr({\Sigma }_r+{\Sigma }_g-2\sqrt{{\Sigma }_r{\Sigma }_g})$$

(25)

where C denotes the vector of the class label, x denotes the original images of the minority class, ${\mu }_r$ and ${\mu }_g$ denote the mean of features, $Tr$ denotes the sum of the diagonal elements, and ${\sum }_r$ and ${\sum }_g$ are the covariance of the feature in the real and generated distribution, respectively.

3.3. Implementation Details

To empirically examine the effectiveness of the proposed IGASEN-EMWGAN model in this paper, extensive experiments are conducted on the real-world dataset: the unbalanced ship coating defects dataset in this section. Apart from it, we slightly fine-tune the network architectures with exiting EGANs in previous works [22,34] to generate controllable and high-quality images of the minority class. The detailed network architectures and the experimental parameters settings of the proposed IGASEN-EMWGAN model in our experiments are distinctly displayed in

Table 3. The architectures of the generator and base discriminator network used in the proposed IGASEN-EMWGAN model.

Generator Network	Discriminator Network
Input: Random noise $z\sim p_z$ (100 dimensions) and Class label C (10 dimensions)	Input: Images, (128,128,3)
[layer 1] Embedding, Dense, BN; Reshape to (8,8,1024) and Concatenate; ReLU;	[layer 1] Conv2D (64,64,128), stride = 2; Dropout; LeakyReLU;
[layer 2] Conv2DT (8,8,1024), stride = 2; ReLU;	[layer 2] Conv2D (32,32,256), stride = 2; BN; Dropout; LeakyReLU;
[layer 3] Conv2DT (16,16,512), stride = 2; ReLU;	[layer 3] Conv2D (16,16,512), stride = 2; BN; Dropout; LeakyReLU;
[layer 4] Conv2DT (32,32,256), stride = 2; ReLU;	[layer 4] Conv2D (8,8,1024), stride = 2; BN; Dropout; LeakyReLU;
[layer 5] Conv2DT (64,64,128), stride = 2; ReLU;	[layer 5] Flatten (1,1,1), Dropout; Dense; Sigmoid/Least Squares;
[layer 6] Conv2DT (128,128,3), stride = 2; Tanh;	[layer 6] Flatten (1,1,1), Dropout; Dense; Softmax;
Output: Generated images, (128,128,3)	Output: Accuracy: Real or Fake (probability); Sample class label C

and

Table 4. The experimental parameters settings of the proposed IGASEN-EMWGAN model.

Hyperparameters	Default Values
Number of iterations	2000
Population size	100
Updating steps of discriminator per iteration	2
Number of variation mutations	2
Number of crossover mutation	1
Probability of variation mutations	0.1
Probability of crossover mutation	0.9
Mini-batch size	64
Learning rate of generator	0.0004
Learning rate of discriminator	0.0001
Dropout	0.5
Slope of LeakyReLU	0.2
Optimizer	RMSProp
Initial learning rate of optimizer	0.0002
Initial annealing temperature	100
Annealing coefficient	0.9

, respectively. All the experiments that have been conducted were repeated five times to add randomness, and all the mean values of the indices are shown in figures and tables.

3.4. Experimental Results and Analysis

3.4.1. Hyperparameters Analysis

In the first experiment, there are two hyperparameters which have an impact on the performance of the proposed IGASEN-EMWGAN model, namely, the balance weight coefficients λ and the number of candidate base discriminators τ. The hyperparameters of SA are also important factors which respond to the performance of IGASEN-EMWGAN. We analyze the effects of setting different numbers of the aforementioned hyperparameters on the performance of the proposed IGASEN-EMWGAN model in this subsection. The details of conducting the related experiments are as follows.

(1)

Balance Weight coefficients λ

In order to select the proper values of the balance weight coefficient λ for IGASEN-EMWGAN, we conduct the experiments on unbalanced ship coating defects database by the strategy of gird search. The Inception Score (IS) evaluation for different IGASEN-EMWGANs under different settings of balance hyperparameters is depicted in Figure 7a. As shown in Figure 7a, we take different balance hyperparameters λ= {0.01,0.1, 0.25,0.5,0.75,1} to balance the quality and diversity of the generated samples. The experimental results demonstrate that a larger setting value of λ (e.g., λ = 0.75) results in slower convergence and the proposed IGASEN-EMWGAN model being out of work at the beginning. Furthermore, a relatively small setting value of λ (λ = 0.1) outperforms the other settings of the balance weight coefficient λ for the sake of the generative performance and the convergence speed. Consequently, we employ an optimal value of the balance weight coefficient λ (λ = 0.1) for the later experiments.

(2)

Number of base discriminators τ

The number of base discriminators has a huge impact on the final generalization performance of the proposed IGASEN-EMWGAN model. If the number of base discriminators is not ideal, the evolved base discriminators will fall into a local optimal solution, and the informative gradient signal provided by discriminators will lack, which results in the lack of an ability to reject generated samples perfectly. Moreover, a larger population of base discriminators often results in the additional time computation cost of the system and the decrease in discrimination performance. To this end, in consideration of the tradeoff between the computational overhead and the discrimination performance, the relationship between the FID evaluation and the number of the base discriminators is shown in Figure 7b. As shown in Figure 7b, as the number of base candidate discriminators τ increases, the value of FID decreases, which means the generative performance of IGASEN-EMWGAN is better. However, the more the base discriminators are ensembled, the computational cost of the system will increase. When the number of base discriminators is eight, the value of FID reaches the minimum value. While adding more base discriminators continuously, the value of FID increases, conversely. To this end, to take the tradeoff between the computational overhead and the improvement of the generative performance, it is suggested that the value of τ is set to be eight. At this time, the median value of FID is 25.413, and the runtime is 64.2 ± 0.8 s.

(3)

The initial temperature T and the annealing coefficient α

In the simulated annealing (SA) algorithm, both the initial temperature T and the annealing coefficient α are the two crucial hyperparameters, as depicted in Algorithm 1. Experiments with different initial temperatures and annealing coefficients are conducted on the real ship coating defect dataset with IR = 107.2. In our experiment, the relationship between the discrimination accuracy of the proposed IGASEN-EMWGAN model and the training epochs under different hyperparameters in SA at different epochs is shown in Figure 8. In our work, we use Grid Search to set different hyperparameters of initial temperatures $T\in \{100,1000,10000\}$ and annealing coefficients $\alpha \in \{0.99,0.999\}$ in SA. As shown in Figure 8, the accuracy of the proposed IGASEN-EMWGAN model increases at a relatively comparable speed at the beginning of training. When the epoch of training is about 750, the accuracy of IGASEN-EMWGAN converges to a certain value, which reveals that the proposed IGASEN-EMWGAN model is robust to different settings of hyperparameters in SA. Furthermore, it can be vividly shown that a smaller setting of the initial temperature T and the annealing coefficient α can make the proposed IGASEN-EMWGAN model converge fast, while a larger setting of the initial temperature T and the annealing coefficient α leads to the opposite result after a certain period. These experimental results show that the proposed IGASEN-EMWGAN model is robust to the different settings of hyperparameters. To this end, the annealing coefficient α and the initial temperature T are recommended to be 0.99 and 1000, respectively.

3.4.2. Comparisons with Different Existing GANs in Generative Performance

In this section, to further demonstrate the superiority of the proposed method, we discuss the relationship between the survived parents’ number and generative performance. The comparative results with various GANs in the real ship coating defect dataset are listed in

Table 5. Comparison among various GANs on the ship coating defect dataset in terms of IS and FID scores. “-” indicates that no results were reported in this paper. (The ↑ means the higher the better, while the ↓ means the opposite.)

Methods	Inception Score (IS) ↑	Fréchet Inception Distance (FID) ↓
Real data	11.68 ± 0.14	7.6
-Standard CNN-
DCGAN [68]	6.52 ± 0.09	36.3
WGAN-GP [34]	6.61 ± 0.33	39.59
E-GAN [22]	6.93 ± 0.08	35.3
AEGAN [70]	6.43 ± 0.52	49.68
LSGAN [71]	-	44.16
EASGAN [72]	7.48 ± 0.06	21.94
E-GAN-GP (μ = 1) [22]	7.18 ± 0.05	32.8
E-GAN-GP (μ = 2) [22]	7.25 ± 0.11	31.4
E-GAN-GP (μ = 4) [22]	7.33 ± 0.08	29.5
E-GAN-GP (μ = 8) [22]	7.35 ± 0.07	27.4
(ours) IGASEN-EMWGAN (τ = 1)	7.11 ± 0.06	32.3
(ours) IGASEN-EMWGAN (τ = 2)	7.18 ± 0.11	29.7
(ours) IGASEN-EMWGAN (τ = 4)	7.31 ± 0.09	27.8
(ours) IGASEN-EMWGAN (τ = 8)	7.56 ± 0.07	26.3
(ours) IGASEN-EMWGAN-GP (τ = 1)	7.21 ± 0.05	30.2
(ours) IGASEN-EMWGAN-GP (τ = 2)	7.33 ± 0.07	28.9
(ours) IGASEN-EMWGAN-GP (τ = 4)	7.68 ± 0.10	26.5
(ours) IGASEN-EMWGAN-GP (τ = 8)	7.73 ± 0.06 1	25.4 2

¹ The bold value indicates the highest value of IS among various GANs, ² The bold value indicates the highest value of FID among various GANs.

. As shown in Table 5, both the metrics of the Inception Score and FID score are employed to evaluate the generative performance of learned generators in our work. Since E-GAN [22] is the most familiar to our method, we further take E-GAN as the baseline to compare with IGASEN-EMWGAN. In addition to this, we also utilize different existing state-of-art GANs including DCGAN [69], WGAN-GP [34], AEGAN [70], LSGAN [71], and EASGAN [72] to compare with the proposed model in this paper. In Table 5, we take a number of various generators μ = {1,2,4,8} for E-GAN and various numbers of discriminators τ= {1,2,4,8} for IGASEN-EMWGAN. We train each model in 150 k generator iterations to conduct experiments. The bold values indicate the highest values of IS and FID among various GANs. Note that the results in Table 5 are the average training results of the five-fold cross-validation method. First, compared with the baseline (E-GAN, μ = 1 without GP), adding the term of the GP norm to optimize the discriminator indeed improves the generative performance during training (e.g., 0.25 improvement for IS and 2.8 decrease for FID when μ is set as 1). This reveals that the term of GP is an effective method for providing smoother gradients to update the generators and stabilize the training process. Furthermore, with the number of base discriminators increasing, the values of ID and FID improve and decrease, respectively, which means the balance of the base generators and discriminators of IGASEN-EMWGAN is well adjusted. These quantitative results in Table 5 show that the proposed IGASEN-EMWGAN model achieves better performance than E-GAN on the unbalanced ship coating defects dataset, which means the proposed method can generate diverse high-quality defect images.

3.4.3. Ablation Study

In this section, we set EGAN as the baseline and add the two-point crossover operator (TP) to the proposed IGANSEN-EMWGAN model for the sake of further verifying the important role of each module in our proposed method. The experimental results are conducted on the real unbalanced ship coating defect dataset, and the two-point crossover operator (TP) on the influence of the values of the average FID score under different iterations is displayed in Figure 9.

As depicted in Figure 9, the IGASEN-EMWGAN model with TP crossover can obtain a lower value of average FID than that without TP crossover when the number of iterations is up to 30 k. Furthermore, the average FID value of the proposed IGASEN-EMWGAN with the two-point crossover operator is slightly lower than the IGASEN-EMWGAN without the two-point crossover operator after 50 k iterations. As vividly depicted in Figure 9, the proposed IGASEN-EMWGAN with the two-point crossover operator yields the lowest FID value. Nevertheless, the process of adding the two-point crossover operator has no considerable influence on the performance of the proposed IGASEN-EMWGAN. A potential reason is that the training epoch is not enough, which leads to the low probability of the crossover operator selection at the very early training stages.

4. Conclusions and Future Work

In this paper, we propose, for the first time, a novel modified NECS and selective ensemble learning driven by the E-GANs, called the IGASEN-EMWGAN model, for generating the new minority images of an unbalanced ship coating defect database without rich training data. The extensive experimental results are conducted on a real unbalanced ship coating defect database and demonstrate that, compared with the baselines and different existing state-of-art GANs mentioned in this paper, the values of the ID and FID scores are significantly improved by 4.92% and decreased by 7.29%, respectively, To this end, in view of these experimental results, we can conclude that the proposed IGASEN-EMWGAN model in this paper can effectively remedy the inherent deficiencies existing in existing GANs and fulfill the valuable image synthesis task successfully when faced with an unbalanced ship coating defect database.

Because the special characteristics of ship coating limit the amount of data, we will consider using online meta learning and transfer learning to reduce the computation cost in future work. In addition, although the genetic algorithm improved by SA can improve the diversity of the generated images to some extent, the global search ability is not powerful, and the operational efficiency is not high. Therefore, the idea of quantum computation theory will be introduced for E-GANs in the future. To justify these hypotheses, we must conduct the related experiments in conjunction with rigorous theoretical analysis, which is another interesting issue for future work.