Image Measurement of Crystal Size Growth during Cooling Crystallization Using High-Speed Imaging and a U-Net Network

Abstract

In this paper, an image measurement method using a high-speed imaging system is proposed for the evolution of crystal population sizes during cooling crystallization processes. Firstly, to resist the negative effect from solution stirring and particle motion during crystallization, a U-net network-based image processing method is established to efficiently detect sufficiently clear crystals from the online captured microscopic images. Accordingly, the crystal size distribution model is analyzed in terms of the counted probability densities of these crystal images. Subsequently, a measurement method of size growth rate based on crystal population distribution is proposed to estimate the growth condition. An experimental case on a crystallization process of β-form LGA is used to show the effectiveness of the proposed strategy.

1. Introduction

Crystallization is an important process to obtain crystalline solids from mixed solutions in pharmaceutical and chemical industries [1]. The cooling crystallization process usually includes the formation of a supersaturated solution, nucleation and crystal growth, etc. Crystal size distribution (CSD) is one of key indicators to evaluate the crystallization production quality [2,3]. It is necessary to measure the growth parameters of the crystal population for process optimization and feedback control [4,5,6]. In recent years, researchers have made significant advances in process analytical technology (PAT) for monitoring crystallization processes, e.g., ATR-FTIR spectroscopy, Raman spectroscopy, focused beam reflectance measurement (FBRM), ultrasound spectroscopy, etc.

With the development of optical imaging sensors, crystallization process detection strategies using image measurement have been promoted for crystal defects, sizes, and shapes [7,8,9,10,11,12]. Larsen et al. [13] analyzed the high concentration crystal image effectively and processed the needle crystal, making a detailed analysis of various characteristics of the crystal from different application values, striving to achieve a comprehensive description of the crystal. Zhou et al. [14] proposed some parameter optimization approaches for image processing applied to extract useful information from microscopy images regarding the distribution monitoring of particle shape and size. Lins et al. [15] developed a detection method of crystal defects, including crystal contour detection and defect quantification for evaluating and optimizing crystallization processes. For the measurement of crystal agglomeration, Ferreira et al. [16] proposed a novel image analysis technique which combined discriminant factorial analysis to assess the agglomeration of crystals. Lu et al. [17] developed a valid crystal segmentation approach based on background difference and local threshold to overcome the negative effect of particle shadow. In previous work [10], an online image measuring method was presented to analyze two-dimension (2D) crystal sizes during cooling crystallization. Gao et al. [18] developed a valid image analysis technology based on deep learning to detect crystals and measure their sizes. Ma et al. [19] applied a novel online image monitoring system to measure the growth rate for mean crystal size during LGA crystallization processes. Traditionally, the growth rate was measured based on the mean sizes for β-form L-glutamic acid [20,21,22,23], but the changes of the crystal population size distribution were not quantized and estimated roundly in the previous literature.

In this work, in order to eliminate the influence of continuous motion in a stirred reactor, an effective strategy is presented for the growth of crystal population sizes using the imaging measurement method based on a deep learning model. Firstly, for the online crystal images influenced by solution turbulence, uneven illumination, and noise, image preprocessing is used. Secondly, the valid crystals are extracted by an effective image segmentation method using a U-net network model. Thirdly, the CSDs are computed with the measured 2D sizes and probability density function. An indicator for describing the growth of crystal 2D size distribution is estimated with a statics method. Experimental results for the case of the cooling crystallization of β-form L-glutamic acid (LGA) show the effectiveness of the proposed imaging measurement method.

The article is organized as follows. Section 2 introduces the basic algorithms. The Section 3 is dedicated to the experimental set-up. Section 4 demonstrates in detail the method of image measurement based on the U-net network. The experimental case is made for the validity of the method in the Section 5. Finally, conclusions are given in the Section 6.

2. Preliminaries

2.1. Classical Convolution Neural Network

Convolutional neural network (CNN) is a basic deep neural network with a convolution structure. In 1998, Lecun et al. [24] designed and trained a CNN model (called LeNet-5), which is a classical CNN structure. The basic structure of CNN is composed of an input layer, a convolution layer, a pooling layer, a full connection layer and an output layer, as shown in Figure 1. Generally, a convolution layer is connected with a pooling layer, and the last few layers near the output layer are usually fully connected networks. The training process of CNN is to learn the convolution kernel parameters of convolution layer and the connection weight between layers. In image recognition, the prediction process is mainly based on the input image and network parameters to calculate the category label.

2.2. U-Net Network

U-net network [25] is a full convolutional network improved based on fully convolutional networks (FCN) [25], which is similar to U-type. Compared with other convolutional neural networks, this network requires smaller training set size and has higher segmentation accuracy. A basic U-net network structure consists of a down-sampling path (encoder) and an up-sampling path (decoder). The down-sampling path is used to obtain the context information, and the up-sampling path is used to pinpoint the location. The down-sampling path is in the left of the network, which consists of 3 × 3 convolution layers and 2 × 2 max pool layers. The activation function $f(x)$ uses ReLU [26], which is defined with $\tau >0$ as

$$f(x)=\left\{\begin{array}{l}x, x>0\\ \tau (e^x-1), x\le 0\end{array}\right.$$

(1)

The up-sampling path is in the right of the network. The deconvolution is used to halve the number of channels, then the deconvolution result is spliced with the corresponding feature map, and the spliced feature map is then convolved with a 3 × 3 kernal. The last layer uses a 1 × 1 convolution to map each 2-bit feature vector to the output layer of the network.

3. Experimental Set-Up

The experimental setup with an imaging system for measuring crystal size distribution is shown in Figure 2. Experiments were carried out with a crystallizer including a 1 L glass jacketed reactor and a PTFE four-paddle agitator. The temperature control device used a Julabo-CF41 thermostatic circulator (JULABO, Seelbach, Germany). A Pt100 temperature probe was used to measure the solution temperature. A camera device (UI-2280SEC-HQ, IDS, Obersulm, Germany) was adopted to record online crystallization images in cooling crystallization processes. The imaging system includes the following functions: image acquisition, image storage, image compression, image output, etc., which was connected to an industrial personal computer. An LED light and controller (Gardasoft -RT260-20, Gardasoft Vision, Cambridge, UK) were employed to provide illumination.

The material used in this experiment was L-glutamic acid (LGA) (Sigma Chemicals, St Louis, MO, USA), which has two forms, the α form and β form. This experiment mainly focused on the study of needle-like β-form LGA. The growth of β-form crystals is statistically analyzed by image analysis in the reactor. The stirring rate was maintained at 200 rpm. Firstly, 0.6 L LGA solution with a concentration of 30 g/L was used in the reactor. The solution was heated to 70 °C and then cooled to 30 °C after 1 h of constant temperature. When the temperature dropped to 55 °C, β-form seeds were added into the reactor, and the online crystal images were recorded with the imaging system.

4. Crystallization Measurement Method

The process of the proposed image measurement method is presented in Figure 3. The detailed steps are described in the following subsections.

4.1. Crystal Image Preprocessing

Due to the disturbance of noise and blur, the quality of crystal images is mostly degraded during online image acquisition. Therefore, guided filtering [27] can be taken as a denoising method. The guided filtering function is considered as an edge-preserving filter, guaranteeing good information preservation around the image edges. Supposing that the pixel of the input image is $m_i$ and the pixel of the output image is $q_i$, which are defined by

$$q_i=\frac{1}{\left|\omega \right|}{\displaystyle {\sum }_{i\in {\omega }_k}{\alpha }_k{m}_i+{\beta }_k}$$

(2)

where $k$ is the index of the local square window ${\omega }_k$ which is taken as $31\times 31$ in the input image, $({\alpha }_k,{\beta }_k)$ are the constants in ${\omega }_k$ which can be obtained by

$$({\alpha }_k,{\beta }_k)=\arg \underset{{\alpha }_k, {\beta }_k}{\min } {\displaystyle \sum _{i\in {\omega }_k}\left({({\alpha }_k{m}_i+{\beta }_k-m_i)}^2+\epsilon {\alpha }_k^2\right)}$$

(3)

where $\epsilon$ is the regularization parameter.

4.2. Crystal Image Segmentation

Since the calibration method uses clear images to compute pixel equivalent, only clear crystals are able to provide accurate size information. In this work, the U-net network model is improved for online crystal images, being also composed of down-sampling path (encoder) and up-sampling path (decoder) [28], as shown in Figure 4. The improved network consists of 11 layer groups. A parametric rectified linear unit (PReLU) [29] is adopted as the activation function to improve the fitting ability. Each Res Block with two 3 × 3 convolutions in the up-sampling path is connected with the corresponding Res Block of the down-sampling path, as shown in Figure 4. The first convolution layer uses a 1 × 1 convolution kernel to extract the features of the input crystal image. Then the designed layer of Res Block is used to further extract image features and deepen the network processing, as shown in Figure 5, which includes the use of two convolutional operations, batch normalization modules and ELU activation functions. After the Res Block in the down-sample path, a 2 × 2 max pooling layer is used to down-sample the feature map, so that the resolution of the feature map decreases to half of the original one. In the up-sampling path, a bilinear interpolation method is used in an up-interpolation block, and the feature map obtained is splicing by layer hopping connection, so that the network can fully fuse the shallow features and deep semantic information. Following the final 1 × 1 convolution layer, the Sigmoid activation function is employed in the last layer to map the response values to (0, 1) pixel by pixel.

To solve the imbalance of positive and negative pixels in the image, the loss function [30] is defined as

$$J_{\mathrm{loss}}=C_{\mathrm{loss}}+D_{\mathrm{loss}}$$

(4)

where $C_{\mathrm{loss}}$ is the cross-entropy loss and $D_{\mathrm{loss}}$ is the Dice coefficient loss.

Image augmentation is used to increase the training sample size and strengthen the network generalization ability. Random rotation, scale, and translation are used to enhance the image diversity, and the brightness and contrast of the image are adjusted to reduce the influence of uneven lighting and highlight the edge features.

4.3. Crystal Growth Measurement

The two-dimensional sizes (i.e., length and width) of β-form LGA crystals are measured based on the length and width of the minimum enclosing rectangle [31] for crystal imaging, respectively. If $l_{\mathrm{a}}$ is the pixel number of length and $w_{\mathrm{a}}$ is the pixel number of width, the pixel equivalent ${\gamma }_{\mathrm{e}}$ is obtained with the calibration method [10]. The physical length $x_l$ and the physical width $x_w$ are given by

$$\left\{\begin{array}{l}{x}_l={\gamma }_{\mathrm{e}}{l}_{\mathrm{a}}\\ x_w={\gamma }_{\mathrm{e}}{w}_{\mathrm{a}}\end{array}\right.$$

(5)

Based on Equation (5), the 2D crystal sizes can be obtained to produce CSD. For a crystal population with a large number of particles showing statistical characteristics, it is meaningful to estimate their size distribution. Generally, the probability density estimation of a log-normal distribution function can be used to smooth the CSD [32] to represent the current size condition of the crystal population.

For length, conforming to the log-normal distribution $\mathrm{LN}({\mu }_l,{\sigma }_l{}^2)$, the likelihood function with length variable $x_l$ is defined as

$$L({\mu }_l,{\sigma }_l{}^2)=\underset{i=1}{\overset{n}{\mathrm{\Pi }}}\frac{1}{\sqrt{2\pi }{\sigma }_l{x}_l(i)}\exp \left\{-\frac{{(\ln x_l(i)-{\mu }_l)}^2}{2{\sigma }_l{}^2}\right\}$$

(6)

The likelihood equations are:

$$\left\{\begin{array}{l}\frac{\partial \ln L({\mu }_l,{\sigma }_l{}^2)}{\partial {\mu }_l}=\frac{1}{{\sigma }_l{}^2}{\displaystyle \sum _{i=1}^n\left(ln{x}_l(i)-{\mu }_l\right)}=0\\ \frac{\partial \ln L({\mu }_l,{\sigma }_l{}^2)}{\partial {\sigma }_l{}^2}=-\frac{n}{2{\sigma }_l{}^2}+\frac{1}{2{\sigma }_l{}^4}{\displaystyle \sum _{i=1}^n{\left(ln{x}_l(i)-{\mu }_l\right)}^2}=0\end{array}\right.$$

(7)

By solving Equation (7), the parameters $({\mu }_l,{\sigma }_l{}^2)$ are estimated as

$${\widehat{\mu }}_l=\frac{1}{n}{\displaystyle \sum _{i=1}^n\ln x_l(i)}$$

(8)

$${\widehat{\sigma }}_l{}^2=\frac{1}{n}{{\displaystyle \sum _{i=1}^n\left(\ln x_l(i)-\frac{1}{n}{\displaystyle \sum _{i=1}^n\ln x_l(i)}\right)}}^2$$

(9)

Then $\mathrm{LN}({\widehat{\mu }}_l,{\widehat{\sigma }}_l{}^2)$ can be computed for denoting the length size distribution of the crystal population by using online images in a predefined time window [8].

The growth parameter of crystal population size is an important factor for crystallization detection. Traditionally, the growth rate in the mean size may not denote the size distribution evolution well, due to the noise of size extremes. Then, $x_l^{\max }$ in the maximum distribution of $P(x_l)$ is computed as

$$x_l^{\max }=\arg \underset{x_l}{\max } P(x_l)$$

(10)

The growth rate of length $R_l$ is defined as

$$R_l=\frac{\mathrm{Diff}(x_{l,t_1}^{\max },x_{l,t_2}^{\max })}{T_{P_1{P}_2}}$$

(11)

where $\mathrm{Diff}$ is the difference function between $x_{l,t_1}^{\max }$ of $P_{t_1}$ and $x_{l,t_2}^{\max }$ of $P_{t_2}$, and $T_{P_1{P}_2}$ is the time interval between point $t_1$ and point $t_2$.

Similar to length, the width size distribution $\mathrm{LN}({\mu }_w,{\sigma }_w{}^2)$ and the growth rate of width $R_w$ are obtained as mentioned above.

5. Experiment Results

5.1. Deep-Learning Crystal Extraction

The experimental case on the crystallization process of β-form LGA was carried out. Compared with traditional methods [8,10,11,33,34], the method of deep-learning crystal extraction did not require too many tedious procedures (e.g., image preprocessing steps or clear crystal identification, etc.). A typically captured image is demonstrated in Figure 6a. Figure 6b displays the result of image preprocessing. In Figure 6c, the crystals are extracted with the deep leaning-based image processing method applied to the image of Figure 6b. It is seen that the minimum enclosing rectangles of the crystals are obtained in Figure 6d. The 2D sizes were obtained from the fitting rectangles. In the experiment, 40 real-time LGA images were employed as the original training set, which uses image augmentation. Flips, translation and rotation were used for variety, and the brightness and contrast were adjusted to reduce the influence of uneven lighting. Finally, noise interference was added to the training set to further improve the generalization ability of the model.

In addition, to represent the superiority of the proposed image processing method, the Ostu segmentation method and the Canny method, which are always used in the image processing of crystals [8,10,11,33,34], were performed for the two crystal images in Figure 7a. Figure 7b demonstrates the results with an improved U-net. Figure 7b,c show the results of image preprocessing using the Ostu and Canny segmentation methods, respectively. It can be seen that clear crystals are detected with an improved U-net, whereas the fuzzy crystals may affect the size measurement. Figure 7 shows that the clear crystals are effectively segmented with the proposed online analysis method, while the other two methods make several segmentation mistakes (e.g., overlapping, fuzziness, etc.). Figure 7e shows that the results of the original U-net model are less accurate than the improved one.

5.2. Crystal Size Measurement

CSD information is important for production. For crystallization quality control, it is necessary that the information feedback of crystal sizes is provided timely and effectively. In the experiment, an offline measurement method using an electric microscope was utilized to verify the accuracy of the CSD measurement with the proposed method. For the same batch of crystals, the comparison study was made between the proposed online method and the offline method by measuring 2D sizes (i.e., length and width), as shown in Figure 8. It is presented that the measured results between the two methods are very similar. It is noted the online images should be captured immediately after the crystals are added into the reactor to avoid the changes of crystal sizes and shapes.

In this experiment, the measured CSDs were fitted by the probability density estimation with the lognormal distribution function. In Figure 9, the CSDs are computed with about 200 crystals collected at time points of 0 min, 20 min, 40 min, and 60 min. The predefined time window was set as 30 s. Figure 9 shows that the crystal population size increases with time, but the range of the crystal size distribution becomes much wider. The needle-like crystals may be easier to be broken by the stirring agitator, resulting in small sizes, whereas the LGA crystals are likely to agglomerate, leading to large sizes. Therefore, the size distribution can become wider at the end of LGA crystallization.

Traditionally, the measurement of growth rate has been used with mean crystal size. However, the growth rate with mean size may not denote the size distribution evolution well, due to the noise of size extremes. It is significant for characterizing the growth rate of crystal size distribution to crystallization control. The evolution of the β-form LGA crystal population size distribution is exemplified in

Table 1. Measured growth rates for 2D crystal population sizes.

Time (min)	${\mathit{R}}_{\mathit{l}}$ (μm/min)	${\mathit{R}}_{\mathit{w}}$ (μm/min)
20 (0–20)	1.69	0.40
20 (20–40)	2.58	0.54
20 (40–60)	3.26	1.11

. At the three intervals of sample times (i.e., 0, 20, 40, and 60 min), the growth rates $R_l$ and $R_w$ were computed by Equation (11) for crystal size distribution. It is observed that the growth rate gradually increases over time, because solution supersaturation is a main driving force for crystal growth in the cooling crystallization. It is also manifested that the growth rates of β-form LGA population length were much larger than those of β-form LGA population width in Table 1. The growth in length is involved in the growth face (101), and the width is related to the faces (010) and (021) [35]. The behavior of length and width growth rates may be in relation to supersaturation, temperature, etc. It is noted that the growth data for length and width are identified more accurately by increasing the number of sampling time points.

6. Conclusions

In this work, an imaging measurement method based on a U-net network was developed to estimate crystal size evolution for β-form LGA using an online non-invasive imaging system. To improve image quality, guided filtering was used for removing the image noise. The deep-learning model with an improved U-net was effectively improved to segment the crystals from the online images. The 2D crystal sizes were measured by using the probability density function. Experimental results showed that the proposed method based on deep learning was effective in obtaining the growth rate of crystal population sizes. The agglomeration condition led to the wrong determination of the crystal sizes. Hence, future work will involve the measurement of crystal agglomeration.