Pixel-Based Approach for Generating Original and Imitating Evolutionary Art

Abstract

We proposed a pixel-based evolution method to automatically generate evolutionary art. Our method can generate diverse artworks, including original artworks and imitating artworks, with different artistic styles and high visual complexity. The generation process is fully automated. In order to adapt to the pixel-based method, a von Neumann neighbor topology-modified particle swarm optimization (PSO) is employed to the proposed method. The fitness functions of PSO are well prepared. Firstly, we come up with a set of aesthetic fitness functions. Next, the imitating fitness function is designed. Finally, the aesthetic fitness functions and the imitating fitness function are weighted into one single object function, which is used in the modified PSO. Both the original outputs and imitating outputs are shown. A questionnaire is designed to investigate the subjective aesthetic feeling of proposed evolutionary art, and the statistics are shown.

1. Introduction

Evolutionary art is a branch of generative art, which is automatically generated by evolutionary computation. Evolutionary computation can be used to create various artworks, including 2D artworks, 3D artworks, music, and animation artworks [1,2]. The genotype–phenotype algorithms, such as genetic algorithms and many nature-inspired evolution algorithms, are widely used to create evolutionary art [3,4]. It is not easy for artificial intelligence to automatically generate artworks with significant aesthetic feeling, since the concept of aesthetic is subjective and hard to measure. In this case, people have proposed some aesthetic measurement methods to guide the algorithms [5,6].

Approaches for evolutionary art can be classified into two categories, agent-based methods and pixel-based methods. The majority of approaches are agent-based. Some of those approaches design a certain number of agents, which are usually few, to act on the canvas with a series of evolution rules [7]. The agents will change the image during the movement. The evolutionary art will be created with the agent’s evolution iterations. Another type of these approaches designed some non-figurative agents that can receive an image as a self-organizing map [8]. The agents evolved individually and the outputs of agents were filtered. The pixel-based methods are less mentioned. The input images were converted into Red-Green-Blue (RGB) or Hue-Saturation-Value (HSV) dataset, and evolved into aesthetically pleasing images. Most of the pixel-based methods are fully automatic evolutionary methods, which do not need manual training or pre-trained data.

In this paper, we propose a pixel-based evolution method for creating evolutionary art. The main motivation of our work is to provide a supplement to existing evolutionary art generation methods. Since there is no significant difference in the essential mechanism between various agent-based methods, the patterns and styles of agent-based evolutionary artworks are similar. There are three advantages of the proposed method. First of all, the artistic style of artworks which are generated by the proposed method is completely different from the artworks generated by agent-based methods. It provides viewers with a different kind of artistic enjoyment. Secondly, the proposed method is a fully automated generation method, which means it does not need any manual intervention, such as data pre-training and parameter adjustment. Thirdly, our proposed method can generate both original artworks and imitating artworks. Taking advantages of its pixel-based characteristics, an imitating fitness function was designed. A target imitating artwork can be input into the algorithm, and the proposed method will generate a special artwork. The special artwork not only imitates the target input but also has its own creative aesthetic features. The main goal of our work is to propose an originally creative, self-organized, fully automated method to generate evolutionary artworks. We employ a modified PSO algorithm to implement the proposed method. The PSO algorithm, known as a kind of automatic evolutionary algorithm, is very suitable for the pixel-based method.

The remainder of this paper is organized as follows. In Section 2, we review the development of evolutionary art. Section 3 describes the proposed approach in detail, including the optimization objects selection and evolution method. Section 4 lists our evolution results, including the original evolutionary artworks and the imitating evolutionary artworks. Section 5 shows the results of the questionnaire about man-made artworks and our evolutionary artworks. The contributions and future works are drawn in Section 6.

2. Background and Related Works of Evolutionary Art

The first application of evolutionary algorithms to create shapes or designs was proposed by Richard Dawkins. The proposed method, named biomorphs, evolved 2D shapes made up of straight black lines. The length, position, and angle of the straight black lines were defined by a set of instructions and rules [9]. After that, Karl Sims successfully applied evolutionary techniques and computer graphics to create evolutionary art with procedural textures. Evolutionary 3D plant structures were grown using genetic parameters. Evolutionary images, animations, and solid textures were created using mutating symbolic lisp expressions [10]. The biomorphs, as some of the earliest evolutionary artworks, were applied to 3D models by Todd et al. The method was called the interactive genetic algorithm, allowing people to manually select the mutant individuals [11]. Ibrahim designed a system named Genshade, which can replicate characteristics of a target image by performing wavelet analysis. The RenderMan shaders with noise functions were evolved, and applied to the Genshade system [12]. The concept of artificial life is suitable for evolutionary art generation. Tiago implemented the artificial life in evolutionary art. In particular, an interactive board was used to let the people directly interact with the agents [13]. There are also many approaches used efficient and autonomous methods such as deep learning, neural networks, and convolutional neural networks [14,15,16].

Aesthetic selection is also a focus of evolutionary art research. Since evolutionary art approaches are automated or semi-automated, they can generate a large number of outputs in a short time. We did not use the word “artworks” because many of the outputs do not have a significant aesthetic value. People face the problem of user fatigue when they manually pick outputs with aesthetic value. Therefore, automated methods to identify artworks are needed. In order to support the exploration of new territory in any generative system, McCormack and Andy visualized both genotype and phenotype space with dimensionality reduction approaches [17]. Easton et al. proposed two approaches to address the user fatigue by improving user engagement, firstly through using virtual environments and secondly improving the predictability of the evolutionary art and giving the user more control [18]. To reduce user fatigue in interactive evolutionary computation with a human-in-the-loop evaluation method, Daniel et al. used the problem of locating interesting fractals to introduce an application of fertility intended. The high fertility sets of fractal parameters are a form of automatically generated content that is part of an application intended to permit a user to find pleasing fractals [19]. Some aesthetic measurement methods and aesthetic learning methods have been proposed to solve the problems [20,21,22].

As we mentioned, there are many agent-based evolutionary art generation approaches. One example of approaches which are driven by figurative agents is Tae and Chang’s work [23]. They proposed a multi-agent-based art production framework for generating artworks that contain highly complex dynamics. Agents acted on the canvas following the three rules of Boids model. Each agent possessed a chaotic neural network which trained by differential evolution algorithm. The colors can be evolved to represent a different style. Another example is Greenfield’s ant painting methods. Greenfield designed a series of ant painting methods, such as ant painting, using a multiple pheromone model [24,25,26]. The ant colony mechanism was employed to generate evolutionary art. Several ants moved on the canvas according to ant colony optimization rules and left colorful trajectories. Different fitness criteria were used to evaluate the aesthetic contributions of the ants and determined different painting styles. Sample outputs of those approaches are shown in Figure 1.

Some state-of-the-art methods have been applied to the evolutionary art generation approaches. Tan et al. introduced the Generative Adversarial Networks (GAN) into the method named ArtGAN [27]. The ArtGAN can synthetically generate artworks that have abstract characteristics, even if the background and foreground of input artworks are not clearly distinguishable. Wang et al. proposed a multimodal convolutional neural network (CNN) that takes into consideration faithful representations of both color and luminance channels, and performs stylization hierarchically with multiple losses of increasing scales [28]. The proposed method solved the texture scale mismatch issue and generated more visually appealing stylized results on high-resolution images. However, compared with our pixel-based approach, they still have some flaws. The CNN-based method needs pre-training data, and therefore it is actually a semi-automated method. Both the ArtGAN method and CNN-based method require an input image to generate artwork, which means that they cannot generate original evolutionary artwork without an input. Our pixel-based approach is a fully automated method, and it can generate original evolutionary artworks without specific inputs.

There are some challenges in evolutionary art. Firstly, many agent-based approaches use genotype–phenotype algorithms, which are very sensitive to the genotype input of the system. A minor change in genotype may produce a radical change in phenotype, leading to an inconsistent genotype and phenotype. In other words, the outputs of genotype–phenotype methods are unstable and unpredictable. Secondly, many agent-based and pixel-based approaches need a fitness function as the target to evolve. The target affects the evolutionary art outputs, so the target should be aesthetically pleasing. The target is a function, which must be mathematically representable and computable. However, as a concept of art and philosophy, aesthetics are still not fully understood by humans. The aesthetic value of an artwork is hard to measure. Another challenge is that fully automated evolutionary art approaches are preferable. As one of the original intentions and ultimate goals of evolutionary art design, the machine will completely autonomously generate evolutionary artworks with aesthetic enjoyment. Many existing approaches require manual intervention or supervision. The main goal of our work is to propose a non-agent-based and automatic approach using aesthetic fitness functions, which can automatically generate evolutionary art with aesthetic enjoyment.

3. Pixel-Based Evolution Method for Creating Evolutionary Art

The proposed approach is mainly based on a modified PSO evolutionary algorithm. To execute the optimization algorithm, a fitness function is necessary. We built a set of aesthetic fitness functions with the Cartesian Genetic Programming (CGP) aesthetic fitness function set, and designed an imitating fitness function [6]. The final fitness function was obtained by weighting two parts. The optimization algorithm we used was modified from the original PSO algorithm. As the evolutionary process was completed, we obtained the output artworks. The flowchart of our approach is shown in Figure 2.

3.1. Optimization Problems Setting for Evolutionary Art

The evolution of evolutionary art needs a certain goal, which can be described as an optimization target. In this section, we proposed several aesthetic fitness functions and an imitate fitness function, and transformed them into a single-objective optimization problem. There are significant differences between the agent-based and the pixel-based approaches for evolutionary art. The agent-based approaches such as genetic algorithm, genetic programming, and ant colony optimization, evolve evolutionary art with few agents (particles). Our pixel-based approach evolves evolutionary art with every single pixel. The pixels keep changing toward a group of certain goals, including the multiple aesthetic fitness and an optional target artwork. This is a standard multi-objective optimization problem which can be described as Equations (1)–(3):

$$\overrightarrow{x}=\left(x_1,x_2,x_h,x_s,x_v\right)$$

(1)

$$\mathit{min} f_h\left(\overrightarrow{x}\right),f_s\left(\overrightarrow{x}\right),f_v\left(\overrightarrow{x}\right),f_t\left(\overrightarrow{x}\right)$$

(2)

$$D=\{\overrightarrow{x}\in R^5|0\le x_1<width, 0\le x_2<height, 0\le x_h<256, 0\le x_s<256, 0\le x_v<256\}$$

(3)

The vector $\overrightarrow{x}$ contains a set of attributes for each pixel. $x_1$ and $x_2$ are constant variable for current pixel. They indicate the coordinates of current pixel and participate in the fitness functions. $x_h$, $x_s$, $x_v$ are the hue, saturation and value (H, S, V) color channels of the current pixel.

The fitness functions $f_h\left(\overrightarrow{x}\right)$, $f_s\left(\overrightarrow{x}\right)$, $f_v\left(\overrightarrow{x}\right)$ are the aesthetic fitness functions. The challenges arise from proposing fitness functions based on aesthetics, since the aesthetics are usually a subjective concept. There have been some aesthetic measurement methods can work out scores of different aspects, to help people to judge evolutionary art fast. However, these methods measure aesthetic with the whole picture. What we want to know is whether the pixels become more “beautiful”. To solve this problem, the aesthetic fitness functions based on CGP are employed. The fitness functions need the coordinates of pixels as inputs, and output the non-negative integer less than 256 as the value of HSV channels. The CGP aesthetic fitness function set has 15 functions, which are shown in

Table 1. CGP aesthetic fitness functions.

${\mathrm{F}}_1$	$(x_1| x_2)\%255$
${\mathrm{F}}_2$	$\left(p\&x_1\right)\%255$
${\mathrm{F}}_3$	$\left(\frac{x_1}{1+x_2+p}\right)\%255$
${\mathrm{F}}_4$	$\left(x_1·x_2\right)\%255$
${\mathrm{F}}_5$	$\left(x_1+x_2\right)\%255$
${\mathrm{F}}_6$	$\left|x_1-x_2\right|\%255$
${\mathrm{F}}_7$	$255-x_1\%255$
${\mathrm{F}}_8$	$\left|255·\cos x_1\right|$
${\mathrm{F}}_9$	$\left|255·\tan \left(\left(x_1\%45\right)·\pi /180\right)\right|$
${\mathrm{F}}_{10}$	$\left|255·\left(\tan x_1\right)\%255\right|$
${\mathrm{F}}_{11}$	$\sqrt{x_1{}^2+x_2{}^2}\%255$
${\mathrm{F}}_{12}$	$x_1\%\left(p+1\right)+255-p$
${\mathrm{F}}_{13}$	$\left(x_1+x_2\right)/2\%255$
${\mathrm{F}}_{14}$	$255·\frac{\left(x_1+1\right)}{\left(x_2+1\right)}, \left(x_1<x_2\right) or 255·\frac{\left(x_2+1\right)}{\left(x_1+1\right)}, else$
${\mathrm{F}}_{15}$	$\left|\sqrt{x_1{}^2-p^2+x_2{}^2-p^2}\%255\right|$

.

The range of the CGP aesthetic fitness functions is $\left[0, 255\right]$. A parameter $p$ was added to increase the flexibility. In order to adapt to the optimization method, it is converted into a single optimization problem (SOP). Our aesthetic fitness functions calculate the absolute value of the difference between the pixel HSV value and CGP aesthetic fitness. The CGP aesthetic fitness functions used in our functions are selected from the CGP function set mentioned above. In particular, a certain function can be selected repeatedly for different channel functions $f_h\left(\overrightarrow{x}\right)$, $f_s\left(\overrightarrow{x}\right)$, and $f_v\left(\overrightarrow{x}\right)$.

$$f_h\left(\overrightarrow{x}\right)=\left|x_h-F\left(x_1,x_2\right)\right|$$

(4)

$$f_s\left(\overrightarrow{x}\right)=\left|x_s-F\left(x_1,x_2\right)\right|$$

(5)

$$f_v\left(\overrightarrow{x}\right)=\left|x_v-F\left(x_1,x_2\right)\right|$$

(6)

The imitating fitness function $f_t\left(\overrightarrow{x}\right)$ is optional. There should be an artwork as the target art. We assume that the resolution of the target art is $W\ast H$. In this case, the coordinate attributes of vector $\overrightarrow{x}$, $x_1$ and $x_2$ are restricted to $x_1\in \left[0,W-1\right]$ and $x_2\in \left[0,H-1\right]$. The HSV values of a pixel for target art are represented by $h$, $s$, and $v$. The imitating fitness function $f_t\left(\overrightarrow{x}\right)$ is described as Equation (7):

$$f_t\left(\overrightarrow{x}\right)={\displaystyle \sum }_{j=0}^{W-1}{\displaystyle \sum }_{k=0}^{H-1}\frac{x_h-h_{ij}+x_s-s_{ij}+x_v-v_{ij}}{3·W·H·\left({\left(x_1-j\right)}^2+{\left(x_2-k\right)}^2+1\right)}$$

(7)

The imitating fitness function describes how similar the current pixel and the target art are. The impact of farther pixels is weakened by a square number of Euclidean geometric distance. In the case where the location of the compared pixel is the current pixel location itself, a baseline of 1 is added. With all of the fitness functions set, the final SOP fitness function for pixel-based evolutionary art is described as Equation (8):

$$\mathit{min} f\left(\overrightarrow{x}\right)=f_h\left(\overrightarrow{x}\right)+f_s\left(\overrightarrow{x}\right)+f_v\left(\overrightarrow{x}\right)+t{f}_t\left(\overrightarrow{x}\right)$$

(8)

The weighting method is the simplest method to integrate the multi-objects into a single object. We assigned the same weight to the aesthetic fitness functions. For the imitating fitness function, we assigned an imitating factor $t$. The higher the imitating factor, the more similar the evolutionary imitating art and the target art are. On the contrary, if we set the value of imitating factor to zero, the optional target art input will be unnecessary, and the method will generate an autonomous evolution art.

3.2. Evolution Method

The pixel characteristic of a photograph is grid-organized. As the genetic algorithm has a lack of topology information, the PSO algorithm is a better choice. The original PSO algorithm has the global neighbor topology. However, it is unnecessary for pixel-based evolution and will result in the loss of local features. In the following part, we use PSO way to refer to things. The “pixels” will be called “particles”; The vector $\overrightarrow{x}$ will be called “location”, since it is a set of values in the search space; The growth of the vector $\overrightarrow{x}$ will be called “velocity”. We modified the global neighbor relationship into a von Neumann topology. The focal particle will only calculate the values of adjacent particles. The modified PSO algorithm is described as Equations (9) and (10):

$$\Delta {\overrightarrow{x}}_i^{k+1}=\omega \Delta {\overrightarrow{x}}_i^k+c_1·r_1·\left({\mathrm{xbest}}_i^k-{\overrightarrow{x}}_i^k\right)+c_2·r_2·\left({\mathrm{vbest}}_i^k-{\overrightarrow{x}}_i^k\right)$$

(9)

$${\overrightarrow{x}}_i^{k+1}={\overrightarrow{x}}_i^k+·{\overrightarrow{x}}_i^{k+1}$$

(10)

where $k$ is the iteration of evolution process and $r_1$, $r_2$ are the random factors in the range of $\left[0, 1\right]$. $c_1$ is the acceleration weight toward particle personal best location. $c_2$ is the acceleration weight toward particle von Neumann neighbors best location. New velocity will be worked out by weighting the personal best location ${\mathrm{xbest}}_{\mathrm{i}}^{\mathrm{k}}$ and von Neumann neighbor best location ${\mathrm{vbest}}_{\mathrm{i}}^{\mathrm{k}}$. The inertial weight factor $\mathsf{\omega }$ controls the impact of the previous velocity on the current velocity. Finally, the new position of the focal particle can be calculated with the old position and the velocity augmenter.

The randomness decides the uniqueness and characteristic of evolutionary art. The randomness of the genotype–phenotype approaches, such as genetic algorithm and genetic programming, comes from the selection-crossover-mutation process. The randomness of PSO approaches comes from the inertia weight $\omega$ and the random factor $r_1$ and $r_2$.

The inertial weight factor $\mathsf{\omega }$ floats with the compactness of swarm. To prevent the particles become premature convergence, a greater inertial weight factor is needed. The inertial weight factor $\mathsf{\omega }$ is self-adaptive and can be calculated by Equation (11).

$$\mathsf{\omega }={\mathsf{\omega }}_{max}-\frac{k}{{\mathrm{iter}}_{max}}\left({\mathsf{\omega }}_{max}-{\mathsf{\omega }}_{min}\right)+r$$

(11)

where $\left({\mathsf{\omega }}_{min}, {\mathsf{\omega }}_{max}\right)$ is the dynamic range of the linear decreasing weight. ${\mathsf{\omega }}_{max}$ is the starting value of inertia weight and ${\mathsf{\omega }}_{min}$ is the ending value of inertia weight. The random seed $r$ will be selected within $\left(0,1\right)$. $k$ is the current iteration number and ${\mathrm{iter}}_{max}$ is the maximum number of iterations. By introducing the kinetic energy of the particle swarm, the inertia weight is reduced with a certain buffer, so that inertia weight can be adjusted adaptively with the kinetic energy of the particle swarm. The self-adaptive updating of inertia weight is very beneficial to the improvement of the precision and also enables the algorithm to converge better.

4. Experiment Results

4.1. Original Evolutionary Art without a Target

Before the proposed evolutionary program runs, a set of initial values should be set appropriately. Firstly, we set the parameters of the optimization objects setting phase mentioned in Section 3.1. The resolution of the canvas was set to $300·300$, therefore the coordinate attributes $x_1$ and $x_2$ were restricted to $x_1\in \left[0,299\right]$ and $x_2\in \left[0,299\right]$. The parameter $p$ of CGP aesthetic fitness functions was set to 10. The functions used in our fitness functions were randomly selected as usual. Next, for the evolution method parameters, there are many parameter selection methods for PSO. We set the parameters in PSO as follows. The starting value of inertia weight was 0.9 and the ending value of inertia weight was 0.4. The range of inertia weight was restricted to $\left(0.4, 0.9\right)$. The maximum number of iterations was set to 150. The acceleration factors $c_1$ and $c_2$ were both set to 2. It has been proven that the initial parameter values we used grant convergence and good exploration, and are still used in many state-of-art optimization algorithms [29].

Figure 3 shows the results of five independent runs. Each row shows the evolutionary art at iteration 0, 30, 60, 90, 120, 150. For those runs, different initial inputs were set. The final evolutionary art shows the random, fuzzy, and soft features. The feature of randomness resulted from $\mathsf{\omega }$, ${\mathrm{c}}_1$, and ${\mathrm{c}}_2$. The feature of fuzziness resulted from the von Neumann topology of particles, since they learn from each other’s values during the iterations. The aesthetic styles are different because the aesthetic fitness functions were randomly selected in different runs.

4.2. Imitating Evolutionary Art with a Target

We used the same parameters in the imitating runs as the previous original runs, except an additional target art input and the imitating factor $\mathrm{t}=0.2$. Figure 4 shows the results of three independent runs. Every single run has two input graphs, four intermediate generations and one final imitating evolutionary art. Our method does not intend to reproduce the target art precisely. The results demonstrate the tendency to evolve towards the target art, but also retain some original characteristics. Subjectively, we can feel the aesthetic feeling of final imitating evolutionary art and some intermediate generation art.

5. Artistic Questionnaire of Proposed Evolutionary Art

The reactions to the evolutionary art by viewers are important. Although cognitive science and other fields are developing, it is still difficult to quantify the subjective aesthetic feeling of an artwork. There are some methods to evaluate the aesthetics of artworks. Questionnaires and surveys are one of the options. We conducted an artistic questionnaire for exploring perceptions of our evolutionary art in terms of aesthetic appraisal. The population of participants was 73, 36 of them were female and 37 of them were male. The mean age of the population was 22.6 and the Standard deviation of population was 11.7. The questionnaire had three questions for prepared artworks:

• Q1. Do you think this artwork is man-made?
• A1. A binary response about “Yes” or “No”.
• Q2. Please rate this artwork from the perspective of artistic sense.
• A2. A Likert item of 5-point Likert scale.
• Q3. Please describe what this artwork wants to depict. (For imitating artworks only)
• A3. A description of the artwork.

Where Q1 and Q2 were prepared for every artwork. Q3 was prepared for only imitating evolutionary artworks. We randomly picked 10 man-made artworks, 10 original evolutionary artworks, and 10 imitating evolutionary artworks. All 30 pictures displayed to the participants were randomly mixed. For Q3, we manually confirmed whether the answer was correct. The statistical results of the questionnaire are shown in Figure 5. The results of Q1 show that when the three kinds of artworks were fully mixed, the majority of participants thought that the prepared artworks were man-made artworks, even if two thirds of them were actually evolutionary artworks. The results of Q2 show that the imitating evolutionary artworks achieved slightly lower scores than the man-made artworks (Ave = 3.8, Ave = 4.2). For the original evolutionary artworks, the gap of scores was large, the average score was acceptable (Ave = 3.3). The results of Q3 show that the majority of participants can correctly recognize the imitating target of proposed imitating evolutionary art.

6. Our Contributions and Future Works

Based on all the generation results and the questionnaire, we can conclude that the proposed pixel-based method can be used as a strategy to guide evolutionary art. The method has a number of advantages. It is a fully automatic evolutionary method, which does not need manual training. Its weight parameters can be adjusted freely to generate different artworks. There are several possible applications of our work. For example, the proposed method can be used to automatically generate stickers for messaging apps (e.g., telegram stickers). Our method can easily generate multiple sticker sets with consistent art styles. Another possible application is developing mobile phone applications with our proposed method. The applications with our method, like some other evolutionary art generation applications such as Art Done Quick, can bring users a pleasant artistic experience. We summarize the main contributions of this work.

• Our work demonstrates the potential of pixel-based method to generate evolutionary art. We perceive the proposed method not as a simple, improved evolutionary art generation method but rather as a meta-method. The proposed method is a supplement for generating evolutionary art and enriches the style of evolutionary art.
• Our method provides a new approach to generate evolutionary imitating art. The imitating fitness function is designed for the pixel-based generation. The similarity of imitation can be simply controlled by the imitating factor.

In the future, we will clarify the relationship between the initial parameters and output styles. We are going to extend the proposed method. It can be extended in the future to generate more complex and diverse artworks; not only 2D artworks, but also 3D artworks. Other evolutionary algorithms will be used for pixel-based evolutionary art.