Scene-Based Nonuniformity Correction Method Using Principal Component Analysis for Infrared Focal Plane Arrays

Abstract

In this paper, principal component analysis is introduced to form a scene-based nonuniformity correction method for infrared focal plane arrays. The estimation of the gain and offset of the infrared detector and the correction of nonuniformity based on the neural network method with a novel estimation of desired target value are achieved concurrently. The current frame and several adjacent registered frames are decomposed onto a set of principal components, and then the first principal component is extracted to construct the desired target value. It is practical, forms fewer ghosting artifacts, and considerably promotes correction precision. Numerical experiments demonstrate that the proposed method presents excellent performance when dealing with clean infrared data with synthetic pattern noise as well as the real infrared video sequence.

Infrared focal plane arrays (IRFPAs) are widely utilized in night vision detection, industrial monitoring, medical imaging applications, and scientific research. However, due to its inconsistent pixel-level response, it exhibits a noticeable fixed-pattern noise (FPN), which is superimposed on the true image [1]. Furthermore, slow drifts in the infrared detectors’ parameters tend to appear with variation in the transistor bias voltage and environmental conditions, making the traditional reference-based (e.g., two-point calibration [2]) nonuniformity correction (NUC) methods insufficient. So, plenty of scene-based nonuniformity correction (SBNUC) methods, which could be continuously adaptively updated according to scene information, have been developed to overcome this problem. Generally, SBNUC methods can be classified into two categories, i.e., statistical methods [3,4,5,6,7,8,9,10] and registration-based methods [11,12,13,14]. The constant statistics (CS) method proposed by Harris and Chiang [3] first assumes a Gaussian distribution of incident irradiance, in which all pixels display the same mean and variance of the irradiance of a scene. Scribner et al. [4] originally developed the neural network (NN-NUC) method, which adaptively estimates a “desired” value to obtain the nonuniformity parameters. Based on the CS algorithm, the local CS (LCS) algorithm was then proposed by C. Zhang and W. Zhao [7]. On the basis of the Kalman filter, the NUC algorithm was presented by Torres et al. [8]. By introducing a total variation approach, Esteban Vera et al. developed a PDE-based method [9] that adaptively achieves estimations of the gain and bias. Rossi et al. [10] utilized the bilateral filter and presented a de-ghosting technique. For the algorithms based on registration, accurate interframe estimation of the motion is required. Some representatives are the motion-compensated average, [11] the algebraic scene-based algorithm, [12] the interframe registration-based algorithm (IRLMS), [13] and the multiframe registration-based adaptive NUC algorithm (MRA-NUC) [14]. Moreover, the unidirectional total variation model [15], weighted least squares [16], temporal filter [17] as well as spatial correlation [18] have been taken into consideration in order to gain a balance between complexity and real-time performance. Furthermore, with the development of various kinds of neural networks, methods based on deep learning have been gradually introduced into stripe removal and nonuniformity correction [19,20,21]. All the NUC methods face the same problems of convergence speed and “ghosting artifacts”. In this paper, a multiframe registration-based NUC method combined with principal component analysis (PCA-NUC) is proposed to deal with the FPN. The PCA-NUC first employs a phase correlation algorithm [22] to achieve an accurate multiframe registration, and principal component analysis is introduced to eliminate the noise pattern and finally construct the desired output. To gain a faster convergence speed while updating the parameter estimation, the steepest descent algorithm is adopted. Due to its success in data mining and machine learning, the Karhunen–Loeve (KL) transform [23] is utilized to convert several registered adjacent frames into a set of values of linearly uncorrelated variables called principal components. Then, the first principal component is extracted to construct the desired target value. Compared with the existing methods [6,13], the proposed PCA-NUC shows a better noise elimination capability and an apparent visual quality improvement of the corrected images, as well as a faster convergence speed, with barely any ghosting artifacts. The flow chart of the proposed method is shown in Figure 1.

Here, the responses of all the detectors are assumed to be linear, and gain and bias are employed to model the nonuniformity. The observed output of the detectors $Y_{i,j}(n)$ can be written as follows:

$$\begin{gathered} Y_{i,j}(n)=a_{i,j}(n)\cdot X_{i,j}(n)+b_{i,j}(n) \\ \left(i\in [1,M];j\in [1,N]\right) \end{gathered}$$

(1)

where $a_{i,j}(n)$ and $b_{i,j}(n)$ represent the real gain and bias of the $(i,j)th$ detector at a given frame $n$ with a size of $M\times N$, respectively. $X_{i,j}(n)$ represents the real incident infrared photon flux, which is collected by the respective detector. By applying linear mapping to the observed pixel values, NUC is performed to provide an estimate of the true scene value ${\widehat{X}}_{i,j}(n)$ so that the responses of the detectors appear to be uniform. The calibrated output of the detector can be expressed as follows:

$${\widehat{X}}_{i,j}(n)=g_{i,j}(n)\cdot Y_{i,j}(n)+o_{i,j}(n)$$

(2)

where $g_{i,j}(n)$ and $o_{i,j}(n)$ represent the gain and bias of the linear correction model of the $(i,j)th$ detector, respectively. And, the relations between the real and the modeled gain and bias can be expressed as

$$g_{i,j}(n)=\frac{1}{a_{i,j}(n)}$$

(3)

$$o_{i,j}(n)=-\frac{b_{i,j}(n)}{a_{i,j}(n)}$$

(4)

The error equation for the correction coefficient estimate is the following:

$$E_{i,j}(n)=T_{i,j}(n)-{\widehat{X}}_{i,j}(n)$$

(5)

where $T_{i,j}(n)$ is the expectation output.

We can use the phase correlation algorithm [22] to obtain the translation between two adjacent observed frames.

$$\begin{gathered} \left[d_i(n,t),d_j(n,t)\right]=\underset{i,j}{\arg \max }\mathrm{Re}\left\{FF{T}^{-1}\left(\frac{F_{u,v}(n)\cdot F^{*}{}_{u,v}(n-t)}{\left|F_{u,v}(n)\cdot F^{*}{}_{u,v}(n-t)\right|}\right)\right\} \\ \left(t\in [1,K]\right) \end{gathered}$$

(6)

where $K$ represents the number of adjacent frames. $F_{u,v}(n)$ and $F_{u,v}(n-t)$ denote the Fourier transforms of $Y_{i,j}(n)$ and $Y_{i,j}(n-t)$, and $(u,v)$ are the Fourier domain coordinates. Here, any rotation, scaling, or other warping of the images are neglected, and the interframe motion is assumed to consist only of translation. So, we can obtain the registered frames ${\stackrel{⌢}{Y}}_{i,j}(n-t)$ relative to the current frame, which can be represented with

$$\begin{array}{ll}\hfill Y_{i,j}(n)& ={\stackrel{⌢}{Y}}_{i,j}(n-t)\hfill \\ & =Y_{i+d_i(n,t),j+d_j(n,t)}(n-t)\hfill \\ & =FF{T}^{-1}\left\{F_{u,v}(n-t)\cdot e^{-2\pi j\left[u\cdot d_i(n,t)+v\cdot d_j(n,t)\right]}\right\}\hfill \end{array}$$

(7)

Next, we rewrite the ${\stackrel{⌢}{Y}}_{i,j}(n-t)$ as an $S$-dimensional ($S=M\times N$) column vector $\mathbf{I}(n,t)$, and several adjacent frames are utilized to construct an $S\times K$ matrix $I_{n,K}=\left\{I(n,0),I(n,1),\dots ,I(n,K)\right\}$. It is assumed that $K=S$, and the mean of the matrix $I_{n,K}$ is ${\overline{I}}_{n,K}=\frac{1}{K}\cdot {\displaystyle \sum _{k=1}^{K}I(n,k)}$. According to the method in [23], the process can be performed using the singular value decomposition (SVD).

$$I_{n,K}-{\overline{I}}_{n,K}=UD{V}^{\mathrm{T}}$$

(8)

where $D$ is a diagonal matrix with eigenvalues that can be derived from $\left(I_{n,K}-{\overline{I}}_{n,K}\right)\cdot {\left(I_{n,K}-{\overline{I}}_{n,K}\right)}^{\mathrm{T}}$ and are listed in descending order. $U$ and $V$ represent an orthogonal transformation matrix in which the columns are eigenvectors computed from $\left(I_{n,K}-{\overline{I}}_{n,K}\right)\cdot {\left(I_{n,K}-{\overline{I}}_{n,K}\right)}^{\mathrm{T}}$ corresponding to the eigenvalues in D and ${\left(I_{n,K}-{\overline{I}}_{n,K}\right)}^{\mathrm{T}}\cdot \left(I_{n,K}-{\overline{I}}_{n,K}\right)$, respectively. Once the covariance matrix is diagonalized, the principal components can be obtained from $U$. Therefore, we can extract the first principal component ($U_1$) to construct the desired target value $T(n)$, which is the following:

$$T(n)=U_1\cdot U_1^{\mathrm{T}}\cdot \left[Y_{i,j}(n)-{\overline{I}}_{n,K}\right]+{\overline{I}}_{n,K}$$

(9)

By employing the steepest descent algorithm [4], $g_{i,j}(n)$ and $o_{i,j}(n)$ can be adaptively updated via the following equations:

$$g_{i,j}(n+1)=g_{i,j}(n)+\lambda \cdot E_{i,j}(n)\cdot X_{i,j}(n)$$

(10)

$$o_{i,j}(n+1)=o_{i,j}(n)+\lambda \cdot E_{i,j}(n)$$

(11)

where $\lambda$ denotes the iteration step size coefficient.

When evaluating the estimation of nonuniformity parameters in this study, PCA-NUC is studied using a clean IR video sequence with synthetic nonuniformity and is compared with the original NN-NUC ($3\times 3$ spatial average filter is used as suggested in [6]) and IRLMS methods by applying a set of real infrared image data. Since the stochastic steepest decent technique is utilized by all three methods to achieve an optimization of the correction coefficients, the same global step constant $\lambda$ ($\lambda =0.05$) is adopted. The clean infrared data were captured with a properly calibrated $320\times 256$ HgCdTe FPA camera operating in the 3~5 μm range and working at 25 FPS from a tall building. Using a synthetic gain which has a unit-mean Gaussian distribution with a standard deviation of 0.1 and a synthetic offset with a zero-mean Gaussian distribution with a standard deviation of 30, the corrupted video sequence is obtained.

To measure the difference between the true infrared image and the corrected result, we use the metric named root mean square error, which is defined as follows:

$$RMSE=\frac{1}{M\times N}\cdot \sqrt{{\displaystyle \sum _{i=1}^M{{\displaystyle \sum _{j=1}^N\left[{\widehat{Y}}_{i,j}-Y_{i,j}\right]}}^2}}$$

(12)

where $Y_{i,j}$ and ${\widehat{Y}}_{i,j}$ are the true value and the corrected value of the $(i,j)th$ pixel, respectively. $M\times N$ represents the size of the infrared image. As shown in Figure 2, the RMSE evolution of the three tested algorithms is displayed.

The three curves decrease rapidly in the first 50 frames. The red curve is the RMSE result of the PCA-NUC method, in which the result reduced to below 20 within 20 frames, whereas more than 50 frames are needed for the other two methods. In the first 40 frames, the RMSE for the NN-NUC method drops almost as fast as for the IRLMS, but it falls abruptly after that. At the same time, the red curve decreases in a relatively stable manner and its RMSE reaches 6.28 after 50 frames, decreasing consistently around 30% over the IRLMS’s and at least 50% over the NN-NUC’s. Obviously, due to its fast convergence speed and stability with hardly any rebound, the PCA-NUC method presents a superior performance compared to the other two methods.

As is well-known, SVD is a computational expensive operation. Moreover, many practical factors, such as memory hierarchy and operating system planning, must be considered in real systems. In this paper, our primary focus lies in evaluating the operational speed of various methods. The following tests have been carried out on a PC with an Intel(R) Core(TM) i5 processor and a 16 Gbyte RAM as well as on the Visual Studio 2019 platform. As shown in

Table 1. Average CPU time consumed per frame for the three algorithms.

Algorithm	Average CPU Time(s)
NN-NUC	0.006
IRLMS	0.087
PCA-NUC	0.146

, a rough average of the CPU time consumed per frame of the three algorithms is displayed. Due to the higher operational needs of PCA-NUC, such as registration and SVD, it is more time-consuming compared to the other two algorithms.

In the following subsection, PCA-NUC is applied to a set of 14-bit real infrared data, which was acquired using an $320\times 256$ HgCdTe FPA camera operating in the 8~14 μm range. When testing the NUC performance with real infrared data, the roughness index ($\rho$), which measures the high-pass content of an image, can be adopted as a reference for comparison to obtain the calibration data needed to perform a radiometrically accurate correction. The definition of $\rho$ can be written as follows:

$$\rho =\frac{mean(\left|L\ast Y\right|)}{mean(Y)}$$

(13)

where $L$ represents a Laplacian filter; $\ast$ is the discrete convolution operator. $Y$ is the image, and $mean(Y)$ represents the mean value of $Y$. The roughness evolution of the three tested algorithms and the result of the mean roughness over the video sequence are presented in Figure 3 and

Table 2. Mean roughness $\rho$ results for the real IR data.

Algorithm	Roughness $\mathit{\rho }$
Unprocessed	0.2721
NN-NUC	0.2143
IRLMS	0.1998
PCA-NUC	0.1851

, respectively.

As shown in Figure 3, all three methods display a distinct capability of correcting nonuniformity. During whole the sequence, PCA-NUC exhibits the lowest roughness among all three methods. The curves of the other two algorithms drop dramatically in the first 50 frames and fluctuate around the mean value in the remaining frames. The PCA-NUC method presents an excellent performance when compared with the other two methods.

Figure 4 shows frame 30 of the real infrared video sequence. The true scene irradiance is displayed in Figure 4a. The outputs obtained using the NN-NUC, IRLMS, and PCA-NUC methods are shown in Figure 4b–d, respectively. As shown in Figure 4b,c, the clear residual fixed-pattern noise is noticeable. Since the motion of the scene is not sufficient, it is a tough task to correct low spatial frequency nonuniformity effectively. The red arrow shows the residual patches of fixed-pattern noise. The results demonstrate that, within only 30 frames, the FPN was almost eliminated with the proposed PCA-NUC method.

As shown in Figure 5, another sample image (the 140th) of the real infrared data is displayed, in which all three NUC methods eliminate the nonuniformity presented in the raw image. The camera moved in the vertical direction from the bottom–up, then suddenly moved to the right. In the outputs, especially in Figure 5b, ghosting artifacts are noticeable (indicated with the red arrow). Also, the residual low spatial nonuniformity is visible in Figure 5b. There exists some residual nonuniformity that can be perceived in the output of the IRLMS method. However, in the PCA-NUC’s output, neither ghost artifacts nor residual inhomogeneity can be detected by the naked eye.

In conclusion, by introducing principal component analysis, a scene-based nonuniformity correction method for infrared focal plane arrays is proposed in this paper. We use principal component analysis to build an eigenspace representation of several registered adjacent frames and then extract the first principal component to construct the desired target value. Some experiments have been conducted to test the proposed method, and the comparison results between the PCA-NUC, traditional neural network, and IRLMS models demonstrate the superiority of our model to the existing methods. In addition, the PCA-NUC method can avoid undesirable effects, i.e., the “burn-in” problem. We have observed that the proposed method shows a significant fixed-pattern noise elimination capability, an apparent visual quality improvement of the corrected images, as well as almost no residual nonuniformity. However, the PCA-NUC method requires a large computational load, so our future research goal will be focused on further reducing its computational complexity for hardware implementation to improve the model’s real-time performance.