# A Joint Training Model for Face Sketch Synthesis

^{1}

^{2}

^{*}

## Abstract

**:**

## Featured Application

**The proposed face sketch synthesis method can be applied for various applications, such as law enforcement and digital entertainment.**

## Abstract

## 1. Introduction

- (1)
- To consider the training sketches during the reconstruction weight representation process, a joint training model is proposed to integrate the training photo and sketch information.
- (2)
- We design a modified locality constraint that modulates the reconstruction weight through the distance between the high-pass filtered images of test patches and the sampled training sketch patches.
- (3)
- The proposed method yields high quality sketches with more detail information and less noise over the wide range of datasets, promoting the accuracy of the sketch-based suspect identification.

## 2. Technical Backgrounds

**T**, it is divided into patches ${t}^{(i,j)}$ with r pixels overlapping between neighboring patches. $(i,j)$ denotes the location of the patch at the i-th row and the j-th column, $i\in \{1,\cdots ,m\}$, $j\in \{1,\cdots ,n\}$. Notice that each patch is represented as a q-dimensional column vector, where q = p

^{2}, and p is the size of the patch. Similarly, the target sketch is denoted as

**S**. ${s}^{(i,j)}$ denotes the target sketch patch corresponding to the testing patch ${t}^{(i,j)}$. The training dataset, which consists of M photo-sketch pairs, are similarly divided into patches. Let ${X}^{(i,j)}={\{{x}_{k}^{(i,j)}\}}_{k=1}^{K}$ and ${Y}^{(i,j)}={\{{y}_{k}^{(i,j)}\}}_{k=1}^{K}$ denote the set of K selected training photo patches and the corresponding sketch patches of the test photo patch ${t}^{(i,j)}$, respectively. The weight coefficients ${w}^{(i,j)}={({w}_{1}^{(i,j)},\cdots ,{w}_{K}^{(i,j)})}^{T}$ are calculated to linearly combine the candidate sketch patches.

#### 2.1. The LLE Method

#### 2.2. The RSLCR Method

## 3. Joint Training Model for Face Sketch Synthesis

#### 3.1. Joint Training Model

**T**, the high-pass image

**H**is obtained with a LoG filter. Let ${h}^{(i,j)}$ denotes the high-pass image patch corresponding to the test photo patch ${t}^{(i,j)}$. We concatenate these two patches as the joint test patch ${t}_{1}^{(i,j)}$:

#### 3.2. Face Sketch Synthesis

^{2}patches in the search region for one patch location, and there are (2c + 1)

^{2}M joint training photo/sketch patch-pairs. Among these patch-pairs, K joint training photo patches ${U}^{(i,j)}\in {\mathbb{R}}^{2{p}^{2}\times K}$ and joint training sketch patches ${V}^{(i,j)}\in {\mathbb{R}}^{2{p}^{2}\times K}$ are randomly and simultaneously selected.

**1**is a column vector in which all elements are 1. ${\mathrm{C}}^{i,j}=({U}^{(i,j)}-1{t}_{1}^{{(i,j)}^{T}}){({U}^{(i,j)}-1{t}_{1}^{{(i,j)}^{T}})}^{T}$ denotes the data covariance matrix, and $\mathrm{diag}(\cdot )$ extends the vector into a diagonal matrix.

## 4. Evaluation Experiments

#### 4.1. Datasets

#### 4.2. Experimental Setting

#### 4.3. Synthesizesd Sketch Results Comparison

#### 4.4. Face Sketch Recognition

## 5. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Wang, N.; Zhang, S.; Gao, X.; Li, J.; Song, B.; Li, Z. Unified framework for face sketch synthesis. Signal Process.
**2017**, 130, 1–11. [Google Scholar] [CrossRef] - Li, J.; Yu, X.; Peng, C.; Wang, N. Adaptive representation-based face sketch-photo synthesis. Neurocomputing
**2017**, 269, 152–159. [Google Scholar] [CrossRef] - Wan, W.; Lee, H.J. FaceNet Based Face Sketch Recognition. In Proceedings of the 2017 International Conference on Computational Science and Computational Intelligence, Las Vegas, NV, USA, 14–16 December 2017; pp. 432–436. [Google Scholar]
- Zhang, Y.; Wang, N.; Zhang, S.; Li, J.; Gao, X. Fast face sketch synthesis via KD-tree search. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherland, 8–16 October 2016; Springer: Cham, Switzerland, 2016; pp. 64–77. [Google Scholar]
- Wang, N.; Zhu, M.; Li, J.; Song, B.; Li, Z. Data-driven vs. model-driven: Fast face sketch synthesis. Neurocomputing
**2017**, 257, 214–221. [Google Scholar] [CrossRef] - Tang, X.; Wang, X. Face sketch recognition. IEEE Trans. Circuits Syst. Video Technol.
**2004**, 14, 1–7. [Google Scholar] [CrossRef] - Liu, Q.; Tang, X.; Jin, H.; Lu, H.; Ma, S. A nonlinear approach for face sketch synthesis and recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, 20–25 June 2005; pp. 1005–1010. [Google Scholar]
- Song, Y.; Bao, L.; Yang, Q.; Yang, M. Real-time exemplar-based face sketch synthesis. In Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 5–12 September 2014; Springer: Cham, Switzerland, 2014; pp. 800–813. [Google Scholar]
- Wang, X.; Tang, X. Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell.
**2009**, 31, 1955–1967. [Google Scholar] [CrossRef] [PubMed] - Zhou, H.; Kuang, Z.; Wong, K. Markov weight fields for face sketch synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Rhode Island, USA, 18–20 June 2012; pp. 1091–1097. [Google Scholar]
- Gao, X.; Wang, N.; Tao, D.; Li, X. Face sketch–photo synthesis and retrieval using sparse representation. IEEE Trans. Circuits Syst. Video Technol.
**2012**, 22, 1213–1226. [Google Scholar] [CrossRef] - Peng, C.; Gao, X.; Wang, N.; Tao, D.; Li, X.; Li, J. Multiple representations-based face sketch–photo synthesis. IEEE Trans. Neural Netw. Learn. Syst.
**2016**, 27, 2201–2215. [Google Scholar] [CrossRef] [PubMed] - Wang, N.; Gao, X.; Li, J. Random sampling for fast face sketch synthesis. Pattern Recognit.
**2018**, 76, 215–227. [Google Scholar] [CrossRef] [Green Version] - Wang, N.; Gao, X.; Sun, L.; Li, J. Bayesian face sketch synthesis. IEEE Trans. Image Process.
**2017**, 26, 1264–1274. [Google Scholar] [CrossRef] [PubMed] - Zhang, L.; Lin, L.; Wu, X.; Ding, S.; Zhang, L. End-to-end photo-sketch generation via fully convolutional representation learning. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghai, China, 23–26 June 2015; pp. 627–634. [Google Scholar]
- Zhang, D.; Lin, L.; Chen, T.; Wu, X.; Tan, W.; Izquierdo, E. Content-adaptive sketch portrait generation by decompositional representation learning. IEEE Trans. Image Process.
**2017**, 26, 328–339. [Google Scholar] [CrossRef] [PubMed] - Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In Proceedings of the International Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; MIT: Cambridge, MA, USA; pp. 2672–2680. [Google Scholar]
- Jiang, J.; Yu, Y.; Wang, Z.; Liu, X.; Ma, J. Graph-Regularized Locality-Constrained Joint Dictionary and Residual Learning for Face Sketch Synthesis. IEEE Trans. Image Process.
**2019**, 28, 628–641. [Google Scholar] [CrossRef] [PubMed] - Wang, N.; Gao, X.; Sun, L.; Li, J. Anchored neighborhood index for face sketch synthesis. IEEE Trans. Circuits Syst. Video Technol.
**2018**, 28, 2154–2163. [Google Scholar] [CrossRef] - Wang, J.; Yang, J.; Yu, K.; Lv, F.; Huang, T.; Gong, Y. Locality-constrained linear coding for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; pp. 3360–3367. [Google Scholar]
- Zhang, W.; Wang, X.; Tang, X. Coupled information-theoretic encoding for face photo-sketch recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA, 21–25 June 2011; pp. 513–520. [Google Scholar]
- Tang, X.; Wang, X. Face photo recognition using sketch. In Proceedings of the IEEE International Conference on Image Processing, New York, NY, USA, 22–25 September 2002; pp. 257–260. [Google Scholar]
- Martinez, A.; Benavente, R. The AR Face Database; Technical Report; CVC: Barcelona, Spain, 1998. [Google Scholar]
- Messer, K.; Matas, J.; Kittler, J.; Luettin, J.; Maitre, G. XM2VTSDB: The extended M2VTS database. In Proceedings of the International Conference on Audio- and Video-Based Biometric Person Authentication, Washington, DC, USA, 22–24 March 1999; pp. 965–966. [Google Scholar]
- Phillips, P.; Moon, H.; Rauss, P.; Rizvi, S. The FERET evaluation methodology for face recognition algorithms. IEEE Trans. Pattern Anal. Mach. Intell.
**2000**, 22, 1090–1104. [Google Scholar] [CrossRef] - Chen, C.; Tan, X.; Wong, K.K. Face sketch synthesis with style transfer using pyramid column feature. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, NV, USA, 12–15 March 2018; pp. 485–493. [Google Scholar]
- Schroff, F.; Kalenichenko, D.; Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 8–12 June 2015; pp. 815–823. [Google Scholar]

**Figure 2.**Example of face sketch-photo pairs in the CUFS dataset (first two rows) and the CUFSF dataset (last two rows). The first and the third row are face photos and the second and the last rows are corresponding face sketches drawn by the artist.

**Figure 3.**Synthesized sketches on the CUFS dataset by locally linear embedding (LLE) [7], Markov random fields (MRF) [9], Markov weight fields (MWF) [10], random sampling and locality constraint (RSLCR) [13], fully convolutional network (FCN) [15], generative adversarial network (GAN) [17], and our proposed method, respectively. Face photos in the first two rows are from the CUHK student dataset; second two rows are from the AR dataset; and the last two rows are from the XM2VTS dataset, respectively.

Methods | LLE | MRF | MWF | RSLCR | Proposed |
---|---|---|---|---|---|

CUHK | 536.34 | 8.60 | 16.10 | 18.79 | 20.80 |

AR | 496.47 | 8.40 | 15.33 | 19.10 | 19.75 |

XM2VTS | 642.50 | 10.40 | 18.80 | 18.14 | 20.78 |

CUFSF | 1591.95 | 24.25 | 45.20 | 17.66 | 20.17 |

Methods | CUFS | CUFSF | ||||
---|---|---|---|---|---|---|

rank-1 | rank-5 | rank-10 | rank-1 | rank-5 | rank-10 | |

LLE | 38.7 | 63.6 | 72.8 | 11.3 | 26.3 | 36.2 |

MRF | 39.6 | 65.1 | 76.3 | 7.2 | 18.0 | 25.3 |

MWF | 47.3 | 68.0 | 77.5 | 10.6 | 23.6 | 31.5 |

RSLCR | 51.7 | 74.2 | 83.4 | 14.3 | 32.4 | 43.1 |

FCN | 51.4 | 76.3 | 84.0 | 11.4 | 26.4 | 35.2 |

GAN | 52.6 | 78.7 | 84.9 | 9.2 | 24.9 | 35.6 |

Proposed | 65.4 | 85.2 | 90.5 | 21.2 | 43.6 | 52.1 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Wan, W.; Lee, H.J.
A Joint Training Model for Face Sketch Synthesis. *Appl. Sci.* **2019**, *9*, 1731.
https://doi.org/10.3390/app9091731

**AMA Style**

Wan W, Lee HJ.
A Joint Training Model for Face Sketch Synthesis. *Applied Sciences*. 2019; 9(9):1731.
https://doi.org/10.3390/app9091731

**Chicago/Turabian Style**

Wan, Weiguo, and Hyo Jong Lee.
2019. "A Joint Training Model for Face Sketch Synthesis" *Applied Sciences* 9, no. 9: 1731.
https://doi.org/10.3390/app9091731