Printed Texture Guided Color Feature Fusion for Impressionism Style Rendering of Oil Paintings

Abstract

As a major branch of Non-Photorealistic Rendering (NPR), image stylization mainly uses computer algorithms to render a photo into an artistic painting. Recent work has shown that the ex-traction of style information such as stroke texture and color of the target style image is the key to image stylization. Given its stroke texture and color characteristics, a new stroke rendering method is proposed. By fully considering the tonal characteristics and the representative color of the original oil painting, it can fit the tone of the original oil painting image into a stylized image whilst keeping the artist’s creative effect. The experiments have validated the efficacy of the proposed model in comparison to three state-of-the-arts. This method would be more suitable for the works of pointillism painters with a relatively uniform style, especially for natural scenes, otherwise, the results can be less satisfactory.

1. Introduction

Non-Photorealistic Rendering (NPR) is one of the important branches of computer graphics. Unlike realistic painting, which pursues physical accuracy and painting authenticity, non-photorealistic painting (NPP) draws scenes through artistic expression techniques, such as oil paintings, pencil drawings, and cartoons et al, so that the painting effect can convey rich feelings [1]. Among various NPP styles, oil painting is widely favored for its long history and rich expressiveness. It can not only accurately depict the scene, as shown in Figure 1a, but also convey various emotions of the painter through different artistic exaggerations (Figure 1b).

In Figure 2, an example is given to convert a photo by a style automatically extracted from an oil painting of a professional artist: Monet.

In the history of art development, Impressionism is a very important category of painting. This category pays attention to the true description of nature, especially the light and shadow changes, which has played a good transitional role in the transfor-mation of European painting from realism to modernism. Because of these unique artistic pursuits the works of this category have different aesthetic characteristics against any other periods and painting categories in terms of subject matter, color, and composition et al. Therefore, the protection of these paintings has become an urgent issue. However, how to digitize these priceless antique paintings and exhibit them on the Internet is reported as a challenging problem.

Due to the characteristics of Impressionism art and similar techniques presented in many such artworks, it is usually difficult for researchers to fully extract the distinctive features from each painter’s work. To address these shortcomings, we develop an adap-tive brush strokes selection approach to automatically extract a generic texture path of an oil painting image. The extracted texture feature will be used to optimize the color feature by guiding the color feature fusion in the Fourier domain. Eventually, the stylized image will be generated by an inverse Fourier transform. A comprehensive experiment with de-tail analysis is reported.

Overall, the main contributions of this paper can be summarized as follows:

• We proposed an unsupervised printed texture guided color fusion framework, namely PTGCF, for impressionist oil painting style migration, which can fully consider the tonal characteristics and the representative color of the original oil painting.
• We proposed an effective fusion strategy to adaptively transfer the texture information from oil painting image to the nature image in the frequency domain of dominant color component.

The remaining of our paper is organized as follows. In Section 2, we give a brief review of the related work. In Section 3, the proposed PTGCF is elaborated in detail. In Section 4, experiments and analysis are given. Finally, some concluding remarks are drawn in Section 5.

2. Related Work

Image stylization is actually an image rendering process, which transfers the color and texture information from one image to another [2]. Previously, some classical methods have been developed. Efros [3] synthesized the target image by extracting and reorganizing texture samples; Hertzman [4] transformed the existing image style into the target image through image analogy; Ashikhmin [5] transformed the high-frequency texture of the source image into the target image while retaining the coarse scale of the target image; Lee [6] improved Ashikhmin’s algorithm by passing additional edge information; Semmo [7] proposed a dominant color quantization model for transforming images into filtered variants with oil paint appearance.

In the last decades, with the wide applications of deep neural network in other visual perception fields, such as face recognition, object detection and even style transfer of images [1]. Gatys [8] first applied the VGG19 network to style migration in 2015, where the key discovery is that the content and style of convolutional neural network are separated, although the style feature representation of any image can be extracted by constructing a Gram matrix. Because the color after the migration may be severely distorted, Gatys proposed two style migration methods to preserve the color information of the original image [9], including RGB space color histogram matching and L*a*b* space brightness migration.

In Gatys’ approach, the filter pyramid in the VGG network is used as the high-level representation of the image. However, different convolution layers can only capture the connections between each pixel of the image, resulting poor rendering effects due to the lack of the connections to the spatial distribution. In a VGG-based fast style migration model proposed in Johnson [10], the efficiency I greatly improved but the shortcomings remain as in [8,9]. Motivated by above problems, Li et al. proposed to combine the Markov random field (MRF) and convolutional neural network [11], replacing the Gram matrix matching in Gatys’ model with a Markov regularization model to improve the visual rationality of the synthetic image. This method provides an interesting extension for style transfer based on image iteration, but it has one major limitation, i.e., the restrictions on the input image: the style image must be able to be reconstructed by MRF. However, a picture with a strong transmission structure is not suitable for this method. As a result, compared with the original image, the image will have the disadvantage of blurred edges. To sum up, the MRF-based style transfer method will produce good results only when the content image and style image have similar shapes without strong changes of angles and sizes.

Liao [12] has achieved good results by combining deep learning (VGG19) and image analogy (patchmatch) for style transfer. With a pre-trained VGG19 network to extract image features through deep learning, they have extended patch-match from the image domain to the feature domain, effectively guiding the semantic level visual attribute migration. The algorithm lacks the ability of semantic generalization, but it can only transfer similar content.

In Zou et al. [13], a neural style transfer model is proposed for stroke-based image-to-painting conversion. Compared with the violent generation of a large number of strokes, this work can outline a stunning oil painting with a relatively small number of strokes. The core idea used is to fix the network parameters of the rendering module and find the optimized stroke parameters in terms of color, coordinate position, and rotation angle, etc. Although this approach is quite flexible in generating painting results for a variety of different stroke types such as oil painting, watercolor, and square, et al., the optimization process is very time-consuming, making it difficult to implement in practical applications. To solve this drawback, Liu et al. [14] has proposed a fast Paint Transformer network, which can quickly transfer a picture into an oil painting with the full usage of texture features. In this work, instead of using an optimization procedure, the stroke parameters are generated by minimizing the difference between the oil painting and the natural photos for improved effectiveness and efficiency. However, it still has many shortcomings, such as poor capability of supporting complicatedly-curved strokes and poor generation effect of slender strokes. In Deng et al. [15], a transformer-based network, namely stytr ^ 2, is proposed to improve the generalizability. Compared with other deep learning models, this approach has better feature representation ability because it captures the long-term dependence of input images and avoids losing of content and style details. Therefore, it can achieve high-quality stylization in terms of good content structure and rich style patterns.

Although deep learning based methods can produce satisfied style synthetic images, it suffers from low efficiency and relying on a large amount of training data [16]. Some fast style transfer models [10,14] has been used to alleviate the problem of low efficiency, yet they can only carry out model training for specific styles thus will need a lengthy process of parameter tuning. Therefore, according to the characteristics of Impressionist oil painting, we introduce a generic image style migration method, based on the rendering of physical model and texture synthesis, and propose an efficient NPR framework for adaptively fusing the texture and color attributes of the target oil painting image to realize the image stylization of impressionist style. Firstly, an adaptive stroke selection method is introduced to automatically obtain the general texture block of oil painting stylized image. Then, the color and texture information of the source image and stroke are fused in the Fourier domain. By making full use of the color and texture features of oil painting stylized images, the source images can be imitated into target images with any artist’s style, such as Monet, Shura and Van Gogh. In order to select the best parameters to obtain the best rendering performance, the selection of pen block size is discussed, and the influence of different painters’ attributes on stylized images is analyzed.

3. Materials and Methods

3.1. Data Collection and Pre-Processing

The experimental image data are acquired by a D65 Kodak camera with a CMOS sensor of 12 mega pixels. The original digital photos mainly cover the real landscape and portrait in real life. Four examples of acquired data are shown in Figure 3. Due to the inconsistent lighting issue in different scenes, the tone reproduction of each captured images was different, which may cause inconsistent image stylized effects. To solve this problem, a simple yet fast image pre-processing algorithm [17] was used to adjust the chroma and contrast of digital images.

3.2. Workflow of the Proposed PTGCF Model

Figure 4 shows the workflow of the proposed method, which is composed of three major modules, i.e., color space conversion, color matching, and feature fusion. Color space conversion is to convert the image from RGB space to the L*a*b* color space. It is an essential step in color matching and also plays a vital role between color matching and feature fusion. Color matching aims to enhance the original digital images and make its appearance similar to the impressionism oil painting. In feature fusion, the color attributes of enhanced image and brush stroke will be adaptively fused in the L*a*b* color space according to their texture attributes. The reason to choose the L*a*b* color space is because its high capability in simulating the human visual system [18]. Its L component closely matches human brightness perception and can be used to adjust the brightness contrast, a and b components can be modified for precise color balance. These transformations are difficult or impossible in other color spaces such as RGB and CMYK et al., because they are modeled according to the output of a physical device, not human visual perception. The final stylized RGB image will be generated after color space conversion. The implementation of color matching and feature fusion will be detailed below.

3.3. Color Matching

To introduce the impressionism style into the original digital photos, the first step is to imitate the color attributes of the oil painting at the original digital photos. For this purpose, a color transfer between images algorithm [19] is employed. The implementation of color matching [20] is detailed below:

• Input a source and an oil painting image. In the Figure 5, the landscape image on the left is the source image, the middle image is the oil painting image;
• Convert both the source and the oil painting image from the RGB to the L*a*b* color space;
• Calculate the mean value and standard deviation of each channels for the source and oil painting images in the L*a*b* color space, denoted as (${\mu }_{l.s}, {\mu }_{a.s}, {\mu }_{b.s}, {\sigma }_{l.s}, {\sigma }_{a.s}, {\sigma }_{b.s}$) and (${\mu }_{l.o}, {\mu }_{a.o}, {\mu }_{b.o}, {\sigma }_{l.o}, {\sigma }_{a.o}, {\sigma }_{b.o}$);
• Subtract the mean value of each channel (i.e., $l_o$, ${\mathrm{a}}_o$, $b_o$) from the oil painting image in the L*a*b* color space;

  $$l^{\prime }=l_o-{\mu }_{l.o}, a^{\prime }=a_o-{\mu }_{a.o}, b^{\prime }=b_o – {\mu }_{b.o}$$

  (1)
• Scale the oil painting channels by the ratio of the standard deviation of the oil painting image divided by the standard deviation of the source image, multiplied by the channels of oil painting image;

  $$l_{scale}=\frac{{\sigma }_{l.o}}{{\sigma }_{l.s}} l^{\prime }, a_{scale}=\frac{{\sigma }_{a.o}}{{\sigma }_{a.s}} {\mathrm{a}}^{\prime }, b_{scale}=\frac{{\sigma }_{b.o}}{{\sigma }_{b.s}} b^{\prime }$$

  (2)
• Add the mean value of the result of previous step into the source image;

  $$l_{new}=\overline{l_{scale}}+l_s, a_{new}=\overline{a_{scale}}+a_s, b_{new}=\overline{b_{scale}}+b_s$$

  (3)
• Convert the result of previous step from the L*a*b* space to the RGB color space.

3.4. Feature Fusion

After color matching, the texture of enhanced digital images and oil painting images will be analysed, which will be used to further guide the color fusion procedure for producing final optimized image rendering.

3.4.1. Edge Enhancement

For each enhanced digital image, the edge attributes will be extracted in the RGB color space. The USM algorithm from the OpenCV library [21] is employed to enhance and extract the edges. Given a source image $S$, a smoothed image map $G$ is first obtained by using the Gaussian blur. The enhanced image $E$ for $S$ can be calculated by Equation (4), where $\omega \in \left[0.1,0.9\right]$ represents the weight, which is set to 0.6 for fast implementation in this paper.

$$E=\left(S – \omega \ast G\right)/\left(1-\omega \right)$$

(4)

3.4.2. Adaptive Selection of Brush Stroke

Brush strokes refer to the markers produced by the painter in the process of painting, which can be used to describe the characterized objects or to express the subjective emotion. As stated in many literatures [13,22,23], image gradient plays an important role in selecting brush strokes. To consistently imitate the most representative brush stroke from images of the oil paintings, the standard deviation SD(i) in a sliding windows W(i) on the source image is calculated as a gradient map ∇S.

$$SD\left(i\right)=\sqrt{\frac{1}{N}{\displaystyle \sum }_{j=1}^N{\left(o_j\left(i\right)-\mu \left(i\right)\right)}^2}$$

(5)

Here we have $i \in \left[1,I\right]$, I is the number of sliding windows with a size of 128 pixels. $N=x\ast y$, $o_j\left(i\right)$ represents the jth pixels in the ith sliding window, $\mu \left(i\right)$ denotes the mean pixel value in the ith sliding window.

The sliding window with the lowest SD will be selected as brush stroke texture patch. 65 original paintings were studied, including 10 Van Gogh’s landscapes, 12 Van Gogh’s portraits, 15 Monet’s landscapes, 12 Monet’s portraits, 9 Seurat’s landscapes, and 7 Seurat’s portraits. The method has a certain universality. Some representative artworks of each artist are shown in Figure 6.

In order to gather our collection of fine-art paintings, we used the publicly available dataset of “Wikiart paintings” (http://www.wikiart.org/, accessed on 1 January 2022); which, to the best of our knowledge, is the largest online public collection of digitized artworks. Examples of selected brush strokes in this study are shown in Figure 7, where Figure 7a,c,e show the oil painting of “Bend in the Epte River near Giverny” by Claude Monet, “Cypress against a Starry Sky” by Van Gogh and “Models.Detail” by Seurat, respectively. Figure 7b,d,f show the selected brush strokes of the three oil paintings.

3.4.3. Feature Fusion in the Fourier Domain

For an actual oil painting artwork, the brush stroke direction and the gradient information are the most important texture attributes, respectively. As shown in Figure 8, for a single orange, the brush stroke direction and gradient of its surface and surrounding shadows have already shown the uniqueness. Therefore, a piece of oil painting will contain more abstract stroke patterns. Unlike oil painting image, digital image doesn’t have brush stroke direction, but the gradient information is still vital to characterise the texture in the image.

Therefore, we take the full usage of texture information in the selected brush as well as the digital image to guide the adaptive color information fusion of brush stroke and digital image in fully rendering the oil painting style on the digital image. The detailed implementation is presented below.

Given a enhance digital image and a selected brush stroke, they will be converted from the RGB color space to the L*a*b* color space. Then, their L* components will be transferred to the frequency domain using the fast Fourier transformation (FFT).

As seen in Figure 9, a power spectrum for each phase angle can be built after transferring the brush stroke into FFT domain. Then the angle of maximum power can be extracted.

On the other hand, the gradient information of each digital image can be extracted after edge enhancement. For each pixel window in the FFT domain, its value will be updated after shifting the brush stroke with the specific phase angle (Figure 10).

Based on this, we came up with an algorithm to add stroke texture to digital photos to produce oil painting style images. Algorithm adapted to stroke (gradient perpendicular to photo gradient), which is summarized as follows.

• Convert the original photograph into the frequency domain.
• Convert the brush texture patch into the frequency domain to create a filter.
• Blur the original photo.
• Enhance the boundary of image in step 3.
• Acquire the direction information of gradation from the output of step 4.
• Update the value of original photograph by adaptively shifting the brush stroke in FFT domain as shown in Figure 10b.
• Convert the result of step 6 into spatial domain by inverse Fourier transformed.
• Concatenate the result of step 7 with original a* and b*. To this end, the color and texture features of digital image and brush stroke have been successfully fused.
• Finally, the fused result of step 8 will be transferred from the L*a*b* to the RGB color space for displaying the optimised rendering effects.

3.4.4. Selection of Stroke Texture

From the movement of stroke texture, it is naturally associated with a problem. That is, to shift the sliding window to fuse stroke texture with enhanced image in order to make the generated image closer to the oil painting. To choose the best size of sliding window, we have compared the performance using three moving strategies, i.e., $\frac{1}{8}$, $\frac{1}{4}$ and $\frac{1}{2}$ of the patch size in the horizontal and vertical directions, respectively (Figure 11).

The evaluation image is printed on photo (glossy) paper with light black ink in the inkjet printer Epson px-5500. The printing size is 12 cm × 16 cm. Under the high color rendering fluorescent lamp of D50, the brightness of 700 lx is evaluated. There is no limit to the observation distance. A subjective evaluate was done by 30 students aged between 20 and 30. The participants were asked to compare the stylized image with Monet’s collection of paintings.

The evaluation method used is the Rank order [24]. This method is to present many stimuli at the same time, where many participants will rank these stimuli according to certain criteria, where the averaged rank score on each stimulus can be obtained. In our study, participants were asked to rank the three oil painting style images according to the criteria of artist-style-similarity between the rendered images and original oil paintings. After massive subjective evaluation, the average rank score of rendered images can be obtained, which can reflect the rendering performance. The stroke texture block to be applied moves 1/8, 1/4 and 1/2 of the patch size horizontally and vertically, and the three generated images are shown in Figure 12. The scale value representing the degree of compliance with the evaluation benchmark of each stimulus is obtained. For landscape 1 and portrait 1, the selection of 1/4 and 1/2 for the patch size gives the same score. On the other hand, for landscape 2 and portrait 2, the selection of 1/8 for the patch size is not ideal. In this way, if the number of overlapping becomes less, the number of mixing colors becomes less.

4. Results

4.1. Landscape Painting

“Starry Night” is one of the most well-known paintings by Van Gogh (Figure 13). There are three dominant colors: sky blue, yellow, and black. In general, the appearance of stylized image is close to the original oil painting images. However, some elements of the oil painting image (i.e., orange moon and the cluster of starligh) don’t reflect on the rendering effects. This is mainly because the painting consists of various texture such as curved long lines, dash lines, swirl patterns, etc. However, the brush stroke texture patch fails to capture all such information, which indicates that the selected brush stroke can merely imitate the general tone and ignore the less-dominant colours.

Impressionism believes that the reality is a fleeting visual impression, which breaks through the previous painting stereotypes which is the usage of high contrast color. In Haystacks series (Figure 14b), Monet used techniques of juxtaposing solid and dot colors, which can reflect the light change in the scene. Although the image stylization result (Figure 14c) can imitate a brown tone of the oil painting, the effect of light change and shadow is insufficient. This can be possibly mitigated by mathematical modelling of computer graphics.

At the end of the 19th century, French painter Georges Seurat was the first one to propose and practice the stippling and color separation painting method which uses a unique stroke to generate colors by primary colors. “Big Bowl Island Sunday Afternoon” is a representative work of Neo-Impressionism (Figure 15b), where the purple color is mixed by parallelly painting the red and blur colors. In addition, this oil painting is composed of millions of widely distributed color dots, which brings the difficulty of fully imitating its color and texture information.

4.2. Portrait Painting

The similarity between the final rendering effect of oil painting stylization and the oil painting image is high. However, due to the lack of semantics of the image content, the rendering effects on the portraits are far away from our human perception. As shown in the third row of Figure 16. Van Gogh’s paintings are formed by thick and powerful strokes, intense, heavy, strong and distorted color lines. However, when Van Gogh’s powerful stroke texture is applied to the portrait, the performance of the skin, the sticky touch of the clothes and the overall texture of the image are affected, which significantly reduce the visualisation. To tackle with this problem, abstract and high-level sematic features can be extremely useful, which can be achieved by multiple convolutional layers with large perception field and pyramid pooling.

4.3. Key Parameter Analysis

With ω varying from 0.1 to 0.9 at a step of 0.1, the rendering performance of using USM as edge enhancement is presented in Figure 17. The overall trends for the Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR) are quite similar. Specifically, SSIM reaches the peak when ω = 0.4. Meanwhile, with ω = 0.1 and 0.4, PSNR reaches the best. Therefore, ω is selected as 0.4 in the USM model and will be used in the following experiments. On a different note, we also compare the rendering performance using USM and the other two classic edge enhancement methods i.e., Gamma transform and Laplacian enhancement [21]. As seen in

Table 1. Quantitative results of different edge enhancement methods in terms of SSIM/PSNR (higher is better).

	SSIM↑	PSNR↑
USM	0.9954	60.23
Gamma	0.9914	58.84
Laplance	0.9930	58.82

, USM outperforms the other two methods, which indicates that USM is more suitable for the image rendering task. Additionally, we also investigate how the size of the brush stroke affects the final rendering performance. As seen in the

Table 2. Quantitative results of Brush stroke window size comparison in terms of SSIM/PSNR (higher is better).

Size (Pixel)	SSIM↑	PSNR↑
128	0.9954	60.23
192	0.9944	60.09
256	0.9944	60.11

, the size of 128 pixels can give the best SSIM and PSNR. Therefore, we set the window size of the brush stroke as 128 in our study. Compared with the results shown in

Table 3. Quantitative results of different methods in terms of SSIM/PSNR (higher is better) and LPIPS (lower is better).

	SSIM↑	PSNR↑	LPIPS↓
PTGCF	0.9954	60.23	0.0306
FT	0.9914	58.84	0.0312
PLST	0.9930	58.82	0.0327
ASNN	0.9911	58.93	0.0656

, selecting any size of the brush stroke can give us better results than all three compared approaches.

4.4. Objective Comparison

To further validate the effectiveness of our proposed method, three state-of-the-art benchmarking methods are utilized for comparison, including Fast Stylization (FT) [25], Perceptual Losses for real-time Style Transfer (PLST) [10], and Artistic Style Neural Network (ASNN) [8]. Qualitative and quantitative results using the Vincent van Gogh’s painting are shown in Figure 18 and Table 3, respectively.

As seen in Figure 18, the proposed PTGCF method can simulate the color tone of the oil paintings, preserve the artist’s original brush strokes and most importantly, keep the semantic content of images much better than the other three benchmarking methods. For the portrait examples, the person can be clearly recognized in our rendered result. Similarly, for the scene samples, the objects such as trees, houses and the chair can be well recognized in our results. Meanwhile, for the third sample, the features of shadows on the street and house can also be characterized. However, all three benchmarking methods fail to preserve the semantic contents in the rendered images. For quantitative results using PSNR, SSIM and LPIPS [26], it can be clearly seen that our proposed PTGCF can produce higher SSIM and PSNR, but lower LPIPS scores than all the compared approaches, which has further validated its efficacy.

5. Conclusions

In this paper, an effective PTGCF model is proposed for optimized impressionism-focused image stylization, which takes the full use of both the representation color and the brush stroke texture of the oil paintings and produce much improved visual effect in the results. Four source photographs were selected: two landscapes and two portraits, and 65 oil paintings were studied, including 10 Van Gogh’s landscapes, 12 Van Gogh’s portraits, 15 Monet’s landscapes, 12 Monet’s portraits, 9 Seurat’s landscapes, and 7 of Seurat’s portraits. The proposed method has shown its generalization in dealing with various artist styles. For our proposed model, it is easy to learn some Impressionist features, though these are difficult to learn for other algorithms or models, if they focus more on universal oil painting. This may be because the trajectory of texture rendering used in our PTGCF model is based on Impressionism. Meanwhile, within the existing framework, the further improvement can be done by replacing some current components with more advanced techniques. For example, ref. [27] can be used for pre-processing stage, deep learning based color transfer methods [28] can be used for color matching stage, and transformer-based network [29] can be employed for edge enhancement.

According to the current research trends, there are still further improvements that can be made in the process of image stylization, such as how to determine whether the rendered image belongs to the expected style. Deep learning based analysis can be applied, such as the DenseNet [30] model and MTFFNet [31] model. Therefore, our future work is to establish an improved deep learning network to objectively evaluate the classification of rendered images, in order to achieve the approximation between rendered images and real oil paintings.

Another work would focus on semantic segmentation based image stylization. Semantic information about image content is seldom considered in image stylization. Most of the methods extract the low-level formal features of the image, such as the direction field of the segmentation block and the determined position of the strokes. Due to little consideration is given to the content information of the image, this has led to an inability to achieve the desired effect regarding the position and direction of the brush strokes. Moreover, there is a lack of distinction between the important and non-important elements of the image. Stylization, such as various forms of filtering operations, abstracts graphics to a certain extent. Therefore, content-aware image stylization can be a new insight in the future.