High-Quality Computational Ghost Imaging with a Conditional GAN

Abstract

In this study, we demonstrated a framework for improving the image quality of computational ghost imaging (CGI) that used a conditional generative adversarial network (cGAN). With a set of low-quality images from a CGI system and their corresponding ground-truth counterparts, a cGAN was trained that could generate high-quality images from new low-quality images. The results showed that compared with the traditional method based on compressed sensing, this method greatly improved the image quality when the sampling ratio was low.

1. Introduction

Ghost imaging, also known as correlation imaging, has been one of the frontiers and hot spots in the field of quantum optics in recent years [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21]. It is a new imaging technique based on the correlation characteristics of quantum entanglement or classical light field fluctuations. Ghost imaging generally obtains the target object information nonlocally through the intensity correlation between the reference light field and the target detection light field.

Ghost imaging techniques, as new imaging techniques, can solve some problems that are difficult to solve using traditional optical imaging techniques. For example, ghost imaging has a strong anti-interference ability, which leads to it having an imaging quality that is significantly better than that obtained by traditional optical imaging techniques in complex optical environments (such as atmospheric turbulence [22] and scattering mediums [20,21,23]). Recently, ghost imaging techniques have been widely used in laser radar detection [24,25], 3D imaging [26,27,28], X-ray ghost imaging [29,30,31], fluorescence microscopy [32], optical encryption [19,33], and terahertz imaging [34,35,36]. Ghost imaging has also gradually outperformed classical imaging in terms of its extremely weak light imaging ability, its image signal-to-noise ratio, its spatial resolution, and its dynamic range. In addition, researchers have enriched and innovated the technical means of ghost imaging from the perspectives of its irradiation light source, its imaging mechanism, its recovery algorithm, and so on, which has greatly expanded the implementation methods and application scenarios of ghost imaging.

Ghost imaging techniques have attracted much attention due to their significant advantages, such as their strong anti-interference abilities. However, ghost imaging requires a long sampling time and a large number of iterative calculations to obtain high-quality imaging since it needs to conduct multiple measurements. Therefore, it is very important for ghost imaging to obtain an appreciable image quality with fewer measurements.

The development of compressed sensing in signal processing [37,38], which introduced compressed sensing into ghost imaging, has allowed ghost imaging to achieve a relatively higher imaging quality with fewer measurements than traditional methods [39]. For example, Katkovnik et al. applied compressed sensing to computational ghost imaging(CGI), which simplified the experimental device and reduced the number of measurements while guaranteeing a certain imaging quality [40]. Along this line, Wenlin Gong and Shensheng Han and others realized compressed sensing super-resolution ghost imaging by introducing a sparse constraint [41,42,43].

In addition to compressed sensing, artificial intelligence was considered for use in ghost imaging due to its various advantages. For example, Barbastathis et al. were the first to propose the training of a deep neural network for a lensless imaging system to recover phase objects [44]. Lyu Meng [45] developed a neural network based on a DNN to optimize the imaging quality under a low sampling rate, and Shimobaba [46] proposed a neural network based on a U-net structure to achieve the denoising optimization of CGI. Subsequently, the authors of [47] employed a CNN to further improve the image quality of ghost imaging. Zhang [48] proposed a singular-value decomposition compressed ghost imaging method based on deep unfolding to achieve a good antinoise performance at low sampling rates. Although compressed sensing has been applied in ghost imaging, its computational complexity is still relatively high since it involves a large number of iterative calculations. The aforementioned artificial intelligence methods have complex network structures, resulting in a huge number of parameters that need to be optimized.

Against this background, we combined a conditional generative adversarial network (cGAN) [49,50] with CGI to reduce the complexity in the reconstruction computation. Our aim of this work was to be able to quickly and accurately reconstruct a target image with a low sampling ratio. Specifically, first, a low-quality image was obtained through CGI at a lower sampling rate, which was used as the condition input of a lightweight cGAN network, and the trained cGAN then generated a high-quality image so as to improve the overall imaging quality performance. In the simulation experiment, we used the MNIST dataset of 50,000 images [51] and their low-quality ghost images to train the network. Through training, the deep neural network could learn the features of the image and make a prediction, in addition to establishing a mapping between a low-quality image and its repective high-quality image. The rest of this work is organized as follows. In Section 2, we describe the considered scenarios and introduce the network structure. In Section 3, we provide the numerical simulation results and verify the effectiveness of the proposed scheme. Finally, Section 4 gives a brief conclusion of this study.

2. Methods

In this section, the computational ghost imaging system and network structure are summarized, and the network training is then described.

2.1. Imaging Scheme

Differently from traditional ghost imaging, CGI reconstructs an image of an object by correlating the preset speckle light field distribution and the light intensity obtained by the bucket detector. Therefore, it only needs one bucket detector to recover the image of an object. The modulation of the speckle light field can be realized by a light source modulator, such as spatial light modulator (SLM) or a digital micromirror device (DMD). A series of speckle light field designs have been achieved by programming. Specifically, the light source modulator is loaded according to the preset cycle sequence and the target scene is modulated N times in the spatial light field. After the emitted laser is modulated by the light source modulator, the speckle light field for ghost imaging is obtained. The detection value $B_i$ of the corresponding bucket detector can be obtained through a series of illuminations, where the object is illuminated by N different speckle light fields. The $B_i$ can then be expressed as

$$\begin{array}{c}\hfill B_i=\sum _x\sum _y{I}_i\left(x,y\right)T\left(x,y\right),\end{array}$$

(1)

where $T(x,y)$ is the transmittance function of the measured object in which x and y are the coordinates of the object plane and $I_i(x,y)$ is the ith speckle light field, $\forall i\in \{1,2,\dots ,N\}$.

For the reconstruction image of the measured object, the signal intensity $B_i$ detected by the bucket detector and the speckle light field intensity $I_i(x,y)$ can be subjected to a second-order correlation operation to recover the image, which can be expressed as [52]

$$\begin{array}{c}\hfill O\left(x,y\right)=\frac{1}{N}\sum _{i=1}^N\left(B_i-\langle B_i\rangle \right)I_i\left(x,y\right),\end{array}$$

(2)

where $\langle ·\rangle =\frac{1}{N}{\sum }_i·$ denotes the ensemble average over all N different speckle light fields.

We observe from Equation (2) that N speckle light fields are required to restore high-quality target images. However, the value of N is generally relatively large, which causes imaging process to suffer from high computational complexity. In order to overcome this difficulty, we first reduced the number of measurements needed to obtain a relatively low-quality reconstruction image, and we then employed a cGAN to enhance the image quality. The corresponding reconstruction process can be expressed as

$$\begin{array}{c}\hfill \tilde{O}\left(x,y\right)=\mathcal{R}\left\{O\left(x,y\right)\right\},\end{array}$$

(3)

where $\mathcal{R}\{·\}$ represents the cGAN that maps the low-quality image $O\left(x,y\right)$ to its respective high-quality image. Here, we proposed the training of a feasible neural network ${\tilde{\mathcal{R}}}_{cGAN}$, which is given by

$$\begin{array}{c}\hfill {\tilde{\mathcal{R}}}_{cGAN}=arg\underset{G}{min}\underset{D}{max}{\mathcal{L}}_{cGAN}\left(G,D\right)+\lambda {\mathcal{L}}_{L_1}\left(G\right),\end{array}$$

(4)

where G is a generative model and D is a discriminative model. Both G and D are nonlinear mapping functions. The function ${\mathcal{L}}_{cGAN}$ is the loss function of the cGAN. The notation $arg\underset{G}{min}\underset{D}{max}{\mathcal{L}}_{cGAN}\left(G,D\right)$ means that G tries to minimize ${\mathcal{L}}_{cGAN}$ while D tries to maximize it, and arg is the abbreviation of argument. The last item makes the generator near the ground-truth output in an $L_1$ sense. Here, $\lambda$ is a constant and its value is 100. The two loss functions in Equation (4) can be described in detail as follows [49]:

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{cGAN}\left(G,D\right)=\mathbb{E}\left[logD\left(x\mid y\right)\right]+\mathbb{E}\left[log\left(1-D\left(x\mid G\left(z\mid y\right)\right)\right)\right]\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{L_1}\left(G\right)=\mathbb{E}\left[\parallel y-G\left(x\mid z\right){\parallel }_1\right],\hfill \end{array}$$

(6)

where x, y, and z represent the output image, the ground-truth image, and the input image, respectively. Here $\mathbb{E}$ stands for the operation of mathematical expectation. $L_1$ means $L_1-norm$.

2.2. Simulation

Figure 1 is a diagrammatic sketch of computational ghost imaging. The simulation was based on this standard computational ghost imaging setup [39]. Monochromatic light with a wavelength of 532 nm was emitted by the laser source. The phase modulation masks were generated by a computer-controlled SLM according to the following formula:

$$\begin{array}{cc}\hfill & M_r\left(k_1,k_2\right)=exp\left(j2\pi {\phi }_r\left(k_1,k_2\right)\right)\hfill \\ \hfill & 1⩽k_1⩽N_1,1⩽k_2⩽N_2,r=1,2,\cdots ,K,\hfill \end{array}$$

(7)

where the random phase ${\phi }_r\left(k_1,k_2\right)$ is uniformly distributed over the interval $[0,1)$ and is independent for all $k_1$, $k_2$, and r. In the modeling of wavefield propagation, the F-DDT technique was applied [53]. The bucket detector measured the total intensity of the light illuminating it. Considering the noise of the bucket detector when observing the light intensity and the thermal noise of the detector itself, Gaussian white noise was added to the output signal of the bucket detector during the simulation. Finally, the image was reconstructed by calculating the correlation between the measurement intensity of the bucket detector and the corresponding speckle pattern generated by the SLM.

2.3. Network Structure

Inspired by a cGAN [49] and pix2pix [50], we used a neural network whose structure is schematically shown in Figure 2. The inputs of the generator were the low-quality reconstructed images from the CGI system. The output images of the generator and their corresponding ground-truth images constituted a pair of inputs to the discriminator. The output of the discriminator was a binary value, which indicated whether the input generated image was true or fake. The architecture of the generator is shown in Figure 3, and it was based on U-Net [54]. There were only five layers of encoders and four layers of decoders, which was lightweight for our scheme. Each convolution block in the encoders was composed of a convolutional layer with a $3\times 3$ convolutional kernel and the rectified linear unit (ReLU). In addition, a $2\times 2$ convolution kernel was used for the maximum pooling layer during the downsampling process. A $2\times 2$ convolutional kernel was applied in the process of upsampling. For the discriminator we employed a PatchGAN discriminator to produce high-quality target images.

2.4. Preparation of Training Data and Network Training

We recall that the inputs of the training network are pairs of picture data, which are composed of ground-truth images and their corresponding CGI reconstruction results. According to Equation (2), we found that the reconstruction of the target image was, mathematically, the inner product between the measured intensity and the corresponding speckle pattern. Thus, we used the simulation method to generate training data pairs. Specifically, the train set was generated from the MNIST handwritten digit databases through ghost imaging simulation.

The training step was 1000. The program was implemented using Python version 3.6 and the cGAN was implemented based on the TensorFlow framework. The GPU-chip NVIDIA 1080ti was used to accelerate the calculation.

3. Results and Discussions

In this section, we evaluate the performance that was achieved by our improved CGI scheme based on a cGAN. To demonstrate the benefit of our improved scheme, we compared our scheme with traditional CGI and compressive sensing computational ghost imaging (CSCGI) techniques. For the CGI, the correlation between the measured intensity of the bucket detector and the random patterns were calculated to reconstruct the image. For the CSCGI, we further applied the BM3D algorithm on the basis of CGI to reconstruct the target image. In the simulations, each MNIST handwritten digit was binarized and resized to a size of $32\times 32$, which indicated that $N=1024$. In addition, the sampling ratio $\beta$ was set as $\frac{M}{N}$, where $M\in \{4,8,16,32,64,128,256,512,768,1024\}$. Then, we had $\beta \in \{0.39\%,0.78\%,1.56\%,3.12\%,6.25\%,12.5\%,25\%,12.5\%,25\%,50\%,75\%,100\%\}$.

3.1. Results

In Figure 4, the simulation results achieved by our developed scheme are compared with CGI and CSCGI for different values of the sampling ratio, $\beta$. As expected, we first observed that the quality of the target images achieved by all schemes improved with the different values of the sampling ratio, $\beta$. In addition, we observed that the image quality achieved by CSCGI was better than that achieved by CGI when the value of $\beta$ was relatively large (e.g., $\beta \ge 6.25\%$), which is due to the fact that the BM3D algorithm was employed to improve the reconstruct performance in the CSCGI.

We also observed that the image quality achieved by our scheme was superior to that of the CGI and CSCGI schemes, especially when the value of $\beta$ was relatively small (e.g., $\beta \le 6.25\%$). To illustrate this phenomenon, we plotted Figure 5, where the sampling ratio $\beta$ was set as $1.56\%$. From this figure, we observed that the reconstructed image achieved by the CGI and CSCGI schemes were completely overwhelmed by noise, while our proposed scheme still achieved recognizable image quality. This result demonstrates that our scheme can effectively suppress noise.

3.2. Discussions

In this subsection, we employ the root of mean square error (RMSE), the peak signal-to-noise ratio (PSNR), and the structural similarity index (SSIM) to quantify the reconstruction performance. The RMSE is defined as

$$\begin{array}{cc}\hfill & RMSE\left(U,V\right)=\sqrt{\frac{1}{HW}\sum _{i=1}^H\sum _{j=1}^W{\left(U_{i,j}-V_{i,j}\right)}^2},\hfill \end{array}$$

(8)

where H and W are the height and width of the image, respectively. The values U and V are the evaluated image and the reference image, respectively.

The PSNR is defined as

$$\begin{array}{cc}\hfill & PSNR\left(U,V\right)=20{log}_{10}\frac{MAX}{RMSE\left(U,V\right)},\hfill \end{array}$$

(9)

where $MAX=255$ is the maximum gray value of the image.

The SSIM is described in [55] as

$$\begin{array}{cc}\hfill & SSIM\left(U,V\right)=\frac{\left(2{\mu }_u{\mu }_v+C_1\right)\left({\sigma }_{uv}+C_2\right)}{\left({\mu }_u^2+{\mu }_v^2+C_1\right)\left({\sigma }_u^2+{\sigma }_v^2+C_2\right)},\hfill \end{array}$$

(10)

where $\mu$ and $\sigma$ are the mean and variance of U and V, respectively, and ${\sigma }_{uv}$ is the covariances of U and V. The constants $C_1$ and $C_2$ are included to avoid instability when ${\mu }_u^2+{\mu }_v^2$ and ${\sigma }_u^2+{\sigma }_v^2$ are very close to zero. Specifically, we chose $C_1={\left(K_{1}L\right)}^2$ and $C_2={\left(K_{2}L\right)}^2$, where $K_1$ and $K_2$ are tiny constants and L is the dynamic range of pixel values. In this work $K_1=0.01$, $K_2=0.03$, and $L=255$, so we obtained $C_1=6.5025$ and $C_2=58.5225$.

In Figure 6, we evaluate the SSIM, the PSNR, and the RMES performances achieved by CGI, CSCGI and our developed scheme. We note that all the original images in the test set of the MNIST dataset were selected to verify the performance achieved by all schemes. We observed from Figure 6 that both the SSIM and the PSNR performances achieved by all schemes increases with the sampling ratio, while the RMSE performances obtained by all schemes decreased with the sampling ratio. This is because the obtained target image information became richer as the sampling ratio increased. From this figure, we also observed that the SSIM, the PSNR, and the RMSE performances achieved by our scheme were significantly better than that achieved by CGI and CSCGI. Specifically, the SSIM, the PSNR, and the RMSE obtained by our scheme at low sampling ratios (e.g., $\beta =6.25\%$) was even better than that obtained by the CGI and CSCGI schemes at high sampling ratios (e.g., $\beta =100\%$), which showed the superiority of our developed cGAN.

According to $\beta =\frac{M}{N}$ and $N=1024$, this meant $M=64$ when $\beta =6.25\%$, and $M=1024$ when $\beta =100\%$. We know that the smaller the number of samples M, the less time is taken. As we know from Figure 6, with the same performance, our scheme took less time than the GI and CSGI schemes. For example, when the M was 64, the GI scheme took 2.3488 s for each image to be reconstructed, while the CSGI scheme, respectively, took 8.3272 s. When the M was 1024, the GI scheme took 24.3536 s for each image to be reconstructed, while the CSGI scheme, respectively, took 124.6102 s. Our scheme predicted that 10,000 images would take 107.7102 s, and each image would take 0.0108 s. Considering the time required for CGI to reconstruct each image before, the actual time for our scheme to reconstruct each image was only 2.3596 s. When the PSNR was about 13.5 dB, our scheme took 2.3596 s, while the GI and CSGI schemes took 24.3536 s and 124.6102 s, respectively. Therefore, our scheme (at a sampling ratio of $6.25\%$) had the same performance as the GI and CSGI schemes (at a sampling ratio of $100\%$), but the time consumption was greatly reduced.

We also present the corresponding standard deviations of the SSIM, the PSNR, and the RMES. Although the corresponding standard deviations of the PSNR and the RMSE obtained by our scheme were close to those of the CGI and CSCGI schemes, the standard deviations of the SSIM obtained by our scheme was significantly smaller than that of the CGI and CSCGI schemes, especially when the sampling ratio $\beta$ was relatively large.

4. Conclusions

We proposed using a cGAN to improve the quality of images produced with CGI. We analyzed the performance of the proposed ghost imaging method under different sampling ratio conditions and compared the results with conventional CGI and CSCGI schemes. Our observations suggested that the proposed method had much better performance in comparison to the other two schemes, especially at low sampling ratios, which indicated that our scheme has the advantages of being strongly antinoise and having a fast computing speed. Moreover, this work made some useful progress towards exploring the practical applications of GI.