Radiometrically Consistent Climate Fingerprinting Using CrIS and AIRS Hyperspectral Observations

Abstract

We introduce a novel spectral fingerprinting scheme that can be used to derive long-term atmospheric temperature and water vapor anomalies from hyperspectral infrared sounders such as Cross-track Infrared Sounder (CrIS) and Atmospheric Infrared Sounder (AIRS). It is a challenging task to derive climate trends from real satellite observations due to the difficulty of carrying out accurate cloudy radiance simulations and constructing radiometrically consistent radiative kernels. To address these issues, we use a principal component based radiative transfer model (PCRTM) to perform multiple scattering calculations of clouds and a PCRTM-based physical retrieval algorithm to derive radiometrically consistent radiative kernels from real satellite observations. The capability of including the cloud scattering calculations in the retrieval process allows the establishment of a rigorous radiometric fitting to satellite-observed radiances under all-sky conditions. The fingerprinting solution is directly obtained via an inverse relationship between the atmospheric anomalies and the corresponding spatiotemporally averaged radiance anomalies. Since there is no need to perform Level 2 retrievals on each individual satellite footprint for the fingerprinting approach, it is much more computationally efficient than the traditional way of producing climate data records from spatiotemporally averaged Level 2 products. We have applied the spectral fingerprinting method to six years of CrIS and 16 years of AIRS data to derive long-term anomaly time series for atmospheric temperature and water vapor profiles. The CrIS and AIRS temperature and water vapor anomalies derived from our spectral fingerprinting method have been validated using results from the PCRTM-based physical retrieval algorithm and the AIRS operational retrieval algorithm, respectively.

1. Introduction

Spectral information from outgoing longwave radiation (OLR) can be used to determine the contributions of individual climate forcing factors to the total change of Top-Of-Atmosphere (TOA) radiation energy. Such information is valuable for the quantification of key climate feedbacks, thus improving the understanding and modelling of climate changes. There are several spaceborne hyperspectral sounders that provide global scale OLR spectral measurements: Atmospheric Infrared Sounder (AIRS), a grating spectrometer on the Aqua satellite, has been in orbit since 4 May 2002; Infrared Atmospheric Sounding Interferometers (IASI) were launched on Metop-A, Metop-B, and Metop-C on 19 October 2006, 17 September 2012, and 7 November 2018, respectively; and two Cross-track Infrared Sounders (CrIS) were launched on Suomi National Polar-orbiting Partnership (NPP) and polar-orbiting satellites of the Joint Polar Satellite System (JPSS-1) on 28 October 2011 and 18 November 2018, respectively. Since 15 November 2017, the Hyperspectral Infrared Atmospheric Sounder (HIRAS) onboard the FY-3D satellite has joined IASI and CrIS as another Fourier transform spectrometer that can provide atmospheric soundings. Those missions together have already achieved more than a decade long global scale observations. The continuity and follow-on to those missions will eventually provide an Earth observation data record that is long enough to allow trend detection for essential climate variables. The accuracy of climate trend detection depends on instrument measurement accuracy, length of the observation time, and magnitude and correlation time of natural variability [1,2,3]. Wielicki et al. [1] estimated that temperature trend uncertainty can be reduced to be less than 0.2 K/decade (95% confidence level) using three decades of measurements from AIRS, IASI, and CrIS, given a 0.085 K natural variability with an autocorrelation time of 2.3 years. Reliable detection of climate trends within a practically and economically reasonable time frame can be achieved by improving the calibration accuracy of those hyperspectral sounder measurements using an International System of Units (SI)-traceable in-orbit reference [1,3]. Therefore, determining how to best assemble existing hyperspectral sounder measurements and develop data records for key climate variables is becoming a critically important research focus.

Long term trends and anomalies of key climate factors, e.g., temperature and water vapor, can be inferred from corresponding OLR spectra. It can be done by using either “retrieve-then-average” or “average-then-retrieve” approaches. For the “retrieve-then-average” approach, atmospheric and water vapor properties need to be retrieved from instantaneous OLR measurements, then the spatiotemporally averaged atmospheric properties can be used to produce anomalies and trends. The “average-then-retrieve” approach uses spatiotemporally averaged trends or anomalies of spectral radiances to derive corresponding trends and anomalies of climate variables. Both approaches provide inverse solutions that connect changes in OLR with corresponding changes in climate variables.

The “retrieve-then-average” approach was used by the AIRS team to develop Level 3 gridded data products to facilitate climate and trending studies [4]. Level 3 products are the spatial and temporal averages of Level 2 products [5]. Operational Level 2 products for hyperspectral sounder missions like AIRS and CrIS are obtained via iterative physical retrieval processes [5,6,7,8]. Smith et al. [9] used the “retrieve-then-average” approach to investigate systematic differences in climate scale temperature records derived from AIRS, CrIS, and IASI spectra. The dual-regression (DR) retrieval method [10,11] was used to obtain instantaneous temperature profiles. Compared to a physical retrieval method, DR greatly reduces the computational cost by avoiding forward radiative transfer calculations in the retrieval process. However, physically-based algorithms can yield averaging kernels needed for error analysis and ensure a better radiometric consistency by finding solutions that fit OLR spectra.

The “average-then-retrieve” approach, also known as “fingerprinting”, originally was developed for detection and attribution of climate signals in the climate model based results [12,13]. Techniques such as the partial radiative perturbation (PRP) method [14] and the radiative kernel-based method, e.g., [15,16,17,18,19], have been used to separate individual climate feedback effects in the OLR spectra simulated by climate models. Leroy et al. [20] tested a spectral fingerprinting method based on the optimal detection technique to extract a carbon dioxide signal, tropospheric temperature signal, stratospheric temperature signal, and water vapor signal from simulated OLR radiance trends of tropical regions under clear sky conditions. Huang et al. [21] later investigated the optimal detection approach for global scale fingerprinting under all-sky conditions. Besides climate-model-based simulation studies, Kato et al. [22,23] simulated OLR spectra using synthetic data sets with atmospheric profiles from the Goddard Earth Observing System Model or Modern-Era Retrospective analysis for Research and Applications (MERRA), and cloud properties from satellite-based observations. They showed that anomalies of atmospheric profiles, surface skin temperatures, and cloud properties can be retrieved from spatially and temporally averaged OLR spectra. These simulation studies demonstrate the feasibility of the OLR spectral fingerprinting approach.

The two approaches are drastically different in terms of the computational cost. For the “retrieve-then-average” approach, performing the retrieval for all sounder field-of-views (FOVs) or field of regards (FORs) of a decadal-scale hyperspectral sounder data record is a computationally intensive process. It also requires additional postretrieval processing including quality control and the spatiotemporal averaging of the Level 2 data. As a comparison, the most time-consuming part of the “average-then-retrieve” approach is the spatiotemporal averaging of the spectral radiances. The computational cost of the “average-then-retrieve” approach can be more than five orders of magnitude less than the “retrieve-then-average” approach. For example, the production of decade-long monthly mean spectral radiances of AIRS over tropical regions can require days on a personal computer. The computational time for performing the spectral fingerprinting on radiance anomalies is negligible (i.e., seconds on a personal computer). It should be noted that the uncertainty of the climate trend analysis depends on the calibration accuracy of the Earth observation instrument. There are ongoing efforts to continuously monitor and improve the calibration accuracy of the hyperspectral sounder Level 1 data. The “average-then-retrieve” approach (i.e., the fingerprinting scheme), once established, can be used to provide fast climate data reprocessing by avoiding the time-consuming process of generating Level 2 data for each sounder FOV or FOR.

Fingerprinting algorithms that follow either the optimal detection approach [20,21] or the regularized regression approach [22,23] rely on the radiative kernel to find the solution. Radiative kernels used in those simulation studies mentioned above are derived using geophysical parameters given from either climate modelling results [20,21] or synthetic databases [22,23]. Although Fan et al. derived the stratospheric temperature change from 10 Years of AIRS and AMSU-A radiances using radiative kernels from climate models [24], they also pointed out the discrepancy between the AIRS radiances and corresponding synthetic radiances calculated using either the GCM modelling or reanalysis results [25]. Therefore, using climate-mode-based radiative kernels for the fingerprinting of real AIRS spectra can be limited by the errors introduced due to the lack of radiance closure, i.e., the change in AIRS radiances simulated from the climate modelling results are not consistent with the observations. The ability to obtain consistency between observations and corresponding fingerprinting results is critical to any real data applications. It is especially important when different OLR measurements from multiple sensors are used to develop long-term climate data records. Instrument calibration errors and spatial and temporal sampling differences in observations can introduce discrepancies between overlapping measurements of different sensors. It is therefore essential to minimize the errors in a spectral fingerprinting algorithm itself. A radiometrically consistent fingerprinting is required to identify potential systematic differences introduced in retrieved geophysical properties by measurement inconsistencies between sensors. Besides radiative kernels, the characterization for fingerprinting uncertainty is also critical to ensure the accuracy of fingerprinting results. However, there is a lack of a well-defined uncertainty estimation for real data fingerprinting applications. The estimation for fingerprinting uncertainty depends on the radiative kernel to be used, the scale of spatial and temporal averaging, and the measurement uncertainty in the OLR spectra. It is important to use a consistent methodology to account for those factors and pay attention to instrument related differences.

An “average-then-retrieve” approach was used by Strow and DeSouza-Machado to derive atmospheric minor gases and ocean skin temperature anomalies from 16 years of real AIRS observations [26]. However, their study was limited to the clear sky subset of AIRS observations. We introduce in this paper a novel fingerprinting approach suitable for deriving climate change signals from hyperspectral sounder observations under all-sky conditions. This approach uses FOV retrieval results from selected hyperspectral sounder observations to build fingerprinting radiative kernels that can be used to process the complete hyperspectral sounder data set of multiple years. Such an approach combines the benefit of using the physical retrieval to establish the radiometric consistency check and the benefit of using the fingerprinting methodology to reduce the computational cost and facilitate the climate trend analysis. To the best of our knowledge, this paper is the first to use a fingerprinting methodology to analyze climate scale hyperspectral sounder observations under all-sky conditions and provide high vertical resolution anomalies of temperature and water vapor profiles from the surface to the stratosphere.

Section 2 introduces the fingerprinting methodology. Details about a novel single FOV physical retrieval algorithm that can be used under all-sky conditions will be presented in Section 3. In addition, the construction of radiative kernels for the characterization of fingerprinting uncertainty using the physical retrieval results is discussed. Section 4 describes the application of the spectral fingerprinting on real CrIS and AIRS data using a consistent radiative kernel. The fingerprinting results are validated dependently using the CrIS retrieval results and independently using the AIRS Level 3 products. The characterization for systematic differences between overlapping measurements of CrIS and AIRS and the comparison between fingerprinting and physical retrieval results are discussed. Section 5 gives a summary and conclusion.

2. Fingerprinting Methodology

The fingerprinting methodology is based on the linear relationship between the change in climate variables $\mathit{\Delta }\mathit{\alpha }$ within specified spatiotemporal scale and the introduced change in top-of-atmosphere (TOA) spectral radiance $\mathit{\Delta }\mathit{d}$, i.e., the climate fingerprint signal.

$$\mathit{\Delta }\mathit{d}=\mathit{S}\mathit{\Delta }\mathit{\alpha }+\mathit{\epsilon },$$

(1)

where $\mathit{\Delta }\mathit{d}$ is the climate spectral fingerprint, $\mathit{\alpha }$ represents an ensemble of climate variables, $\mathit{S}$ is the radiative kernel that characterizes the linear relationship, and $\mathit{\epsilon }$ is the fingerprinting residual term that includes errors due to the linearization of the radiative transfer equation and the spatiotemporal averaging. The climate spectral fingerprints can be constructed from the spatiotemporal average of individual spectral measurements $\mathit{R}$ as $\mathit{\Delta }\mathit{d}=\mathit{\Delta }\overline{\mathit{R}}$. In correspondence, climate variable $\mathit{\alpha }$ is the spatiotemporal average of individual geophysical properties, i.e., $\mathit{\alpha }=\overline{\mathit{X}}$. $\mathit{X}$ represents the surface, atmospheric, and cloud properties within the measurement scope of an individual $\mathit{R}$. $\mathit{R}$ and $\mathit{X}$ can be related using a radiative transfer forward model as

$$\mathit{R}=F\left(\mathit{X}\right)+\mathrm{ℇ},$$

(2)

where $F$ represents the forward model, $\mathrm{ℇ}$ represents the forward model error and the instrument measurement uncertainty. The climate spectral change from one reference state to another can therefore be calculated as

$$\mathit{\Delta }\mathit{d}=\overline{F\left(\mathit{X}+\mathrm{\Delta }\mathit{X}\right)}-\overline{F\left(\mathit{X}\right)},$$

(3)

$F\left(\mathit{X}+\mathrm{\Delta }\mathit{X}\right)$ can be linearized around the reference state $\mathit{X}$ as

$$F\left(\mathit{X}+\mathrm{\Delta }\mathit{X}\right)=F\left(\mathit{X}\right)+\mathit{K}\mathrm{\Delta }\mathit{X}+\mathit{\sigma }\left(\mathit{X}\right),$$

(4)

where $\mathit{K}$ is the Jacobian for the reference state $\mathit{X}$, and $\mathit{\sigma }$ is the nonlinearity term that includes the higher order expansion of $F\left(\mathit{X}+\mathrm{\Delta }\mathit{X}\right)$. Therefore, Equation (3) can be linearized as

$$\mathit{\Delta }\mathit{d}=\overline{\mathit{K}\mathrm{\Delta }\mathit{X}}+\overline{\mathit{\sigma }\left(\mathit{X}\right)}.$$

(5)

The $\overline{\mathit{K}\mathrm{\Delta }\mathit{X}}$ term in Equation (5) can be expanded as

$$\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N{\mathit{K}}_i\mathrm{\Delta }{\mathit{X}}_i=\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N(\overline{\mathit{K}}+\mathit{\delta }{\mathit{K}}_i)\left(\overline{\mathrm{\Delta }\mathit{X}}+\mathit{\delta }{\mathit{X}}_i\right)=\overline{\mathit{K}}\overline{\mathrm{\Delta }\mathit{X}}+\frac{1}{N}{\displaystyle \sum }_{i=1}^N\mathit{\delta }{\mathit{K}}_i\mathit{\delta }{\mathit{X}}_i,$$

(6)

where $\mathit{\delta }{\mathit{K}}_i$ and $\mathit{\delta }{\mathit{X}}_i$ represent the deviation of the Jacobian and the geophysical state vector for one individual observation from the corresponding spatiotemporal mean values, with $\overline{\mathit{\delta }\mathit{K}}=\mathbf{0}$ and $\overline{\mathit{\delta }\mathit{X}}=$ 0. Therefore, Equation (5) can be rewritten as

$$\mathit{\Delta }\mathit{d}=\overline{\mathit{K}}\mathit{\Delta }\mathit{\alpha }+\overline{\mathit{\delta }\mathit{K}\mathit{\delta }\mathit{X}}+\overline{\mathit{\sigma }\left(\mathit{X}\right)}$$

(7)

Comparing with Equation (1), we can see that the radiative kernel S in Equation (1) can be constructed as the spatiotemporal average of individual Jacobians, S = $\overline{\mathit{K}}$. The fingerprinting residual term $\mathit{\epsilon }$ in Equation (1) includes two parts: $\overline{\mathit{\delta }\mathit{K}\mathit{\delta }\mathit{X}}$ is introduced by the space-time averaging residual; and $\overline{\mathit{\sigma }\left(\mathit{X}\right)}$ is the residual term due to the nonlinear nature of the radiative transfer equation.

The widely used optimal detection technique is also used in this paper to determine the magnitude of multiple climate signals. The concept of the optimal detection technique has been well discussed in various papers [12,13,27]. With a prescribed radiative kernel matrix $\mathit{S}$, the generalized least square solution is given as

$$\mathrm{\Delta }\mathit{\alpha }={({\mathit{S}}^T{\Sigma }^{-1}\mathit{S})}^{-1}{\mathit{S}}^T{\Sigma }^{-1}\mathrm{\Delta }\mathit{d},$$

(8)

where $\Sigma$ is the covariance of the residual term $\mathit{\epsilon }$. $\mathrm{\Delta }\mathit{d}$ and $\mathrm{\Delta }\mathit{\alpha }$ are calculated as the difference between spatiotemporally averaged values of one climate state and those of a reference state. In some studies, $\mathrm{\Delta }\mathit{d}$ and $\mathrm{\Delta }\mathit{\alpha }$ take the form of a trend generalized from corresponding time series data [20,24,25,27]. Optimal fingerprinting of trends assumes that anomalies are normally distributed around the underlying trend of climate change. However, the extraction of climate trends is inevitably obscured by natural variability. Natural variability caused by either external forcing such as volcanic eruptions, variation of solar radiation, or internal forcing such as the El Niño-Southern Oscillation (ENSO) and Quasi-biennial Oscillation (QBO), imposes non-Gaussian radiance features on OLR anomalies. Neglect of such non-Gaussian features will introduce errors in the climate trend fingerprinting. How to accurately account for the additional fingerprinting uncertainty due to the non-Gaussian feature of the natural variability still needs to be studied. In this paper, the optimal detection methodology (defined in Equation (8)) is applied on OLR anomalies to derive atmospheric temperature and water vapor anomalies. By using a linear multipattern regression method [3,28], the trends of atmospheric, cloud, and surface properties can then be extracted from the retrieved anomalies, respectively. The advantage of this method is that one can use high quality external proxy data to fit natural variability in the multipattern regression step.

Allen and Tett [13] pointed out the challenge of quantifying the uncertainty covariance $\Sigma$ in optimal fingerprinting. Previous simulation-based studies characterize the uncertainty based on the radiative kernels and the natural variability from climate models or Observing System Simulation Experiments (OSSEs) [20,21,24,25,27]. Estimation for the fingerprinting residual term $\mathit{\epsilon }$ and covariance $\Sigma$ depends on the choice of radiative kernel $\mathit{S}$ and the space and time scale used to obtain $\mathrm{\Delta }\mathit{d}$ and $\mathrm{\Delta }\mathit{\alpha }$. $\mathit{S}$ is constructed using forward radiative transfer model simulations. Instrument dependent measurement errors and potential forward model errors such as errors in molecular spectroscopy of atmospheric species and cloud modelling have to be accommodated in $\Sigma$ to ensure the consistency between $\mathrm{\Delta }\mathit{d}$ and $\mathrm{\Delta }\mathit{\alpha }$ for fingerprinting applications using real data. Details about the construction and validation of $\mathit{S}$ and $\Sigma$ from real observation data of CrIS and AIRS and the accommodation for measurement difference between CrIS and AIRS are going to be discussed in this paper.

3. PCRTM Single FOV Retrieval Algorithm

A single FOV physical retrieval scheme is used to construct radiative kernels and obtain reference state information needed for the climate fingerprinting study. The core part of the algorithm is the principal component based radiative transfer model (PCRTM) that incorporates the calculation for longwave multiple scattering by clouds in the forward simulation [29,30]. The cloud microphysical properties that match single FOV observations can be retrieved simultaneously with other geophysical parameters. Therefore, a well-defined radiative transfer relationship between observations and key geophysical variables including temperature, humidity, and cloud parameters can be directly constructed. More details about the single FOV retrieval algorithm can be found in previous publications [31,32]. The algorithm is different from NASA operational retrieval algorithms for the CrIS and AIRS missions. Both heritage NASA AIRS Level 2 algorithm [5,6] and current NASA operational Level 2 algorithm [7,8] use a cloud-clearing methodology to avoid the complexity of including the cloud scattering calculation in the radiative transfer simulation [33]. However, the consistency between TOA radiance observations and corresponding radiances simulated from the matching geophysical variables cannot be directly checked due to the lack of simulations for cloud scattering in the cloud-clearing algorithms.

We have processed two years of combined observation data sets from CrIS and the Advanced Technology Microwave Sounder (ATMS) using our PCRTM retrieval algorithm. Hyperspectral IR sensors provide high vertical resolution atmospheric sounding, but their measurement sensitivity to atmospheric profiles below clouds decreases as the cloud optical thickness increases. Consequently, the retrieval of temperature and water vapor profiles below clouds using solely CrIS radiances has larger uncertainties than that can be achieved under a clear-sky condition. The uncertainties increase with the cloud optical thickness. MW radiances such as those from the ATMS are not affected by ice clouds and can provide more sounding information below water clouds than IR observations. Therefore, the ATMS observations collocated with CrIS can be used to provide complementary information to reduce the uncertainty in the IR retrieval below clouds. The combined IR+MW retrieval algorithm follows the maximum likelihood iterative scheme. For a given individual spectral measurement of $\mathit{R}$ (defined in Equation (2)), a solution $\mathit{X}$ is given following the general approach of Rodgers [34]:

$${\mathit{X}}_{n+1}-{\mathit{X}}_n={\left({\mathit{K}}^T{\mathit{S}}_y^{-1}\mathit{K}+{\mathit{S}}_a^{-1}\right)}^{-1}\left({\mathit{K}}^T{\mathit{S}}_y^{-1}\left(\mathit{Y}-{\mathit{Y}}_n\right)-{\mathit{S}}_a^{-1}\left(\mathit{X}-{\mathit{X}}_a\right)\right)$$

(9)

where n is the iteration number, ${\mathit{S}}_y$ is the measurement error covariance matrix, ${\mathit{X}}_a$ and ${\mathit{S}}_a$ are the background and covariance of a priori constraint, $\mathit{K}$ is the Jacobian, i.e., the first derivative which defines the sensitivity of the measurement to the input parameters,

$$\mathit{K}={\frac{d\mathit{R}}{d\mathit{X}}|}_{\mathit{X}={\mathit{X}}_n}$$

(10)

The final solution is found when the cost function J, being defined as

$$J\left(\mathit{X}\right)={\left(\mathit{R}-F\left(\mathit{X}\right)\right)}^T{\mathit{S}}_y^{-1}\left(\mathit{R}-F\left(\mathit{X}\right)\right)+{\left(\mathit{X}-{\mathit{X}}_a\right)}^T{\mathit{S}}_a^{-1}\left(\mathit{X}-{\mathit{X}}_a\right)$$

(11)

is minimized. The principal component (PC) analysis is used to reduce the dimension of the CrIS radiance spectra and the geophysical variables.

Table 1. List of number of principal components (PCs) used for the principal component based radiative transfer model (PCRTM) based single field-of-views (FOV) retrieval.

PC Compressed Variables	Number of PCs
CrIS spectra (Long Wave)	50
CrIS spectra (Mid Wave)	40
CrIS spectra (Short Wave)	30
Temperature	20
Water Vapor	15
Surface Emissivity	10
Ozone	10
CO2	2
N2O	2
CO	4
CH4	2

lists the number of PCs used for the retrieval.

The single FOV physical algorithm is used to achieve high vertical and horizontal spatial resolution retrieval of temperature and water vapor under all-sky conditions. The application of single FOV physical retrieval algorithm has been tested using the CrIS/ATMS data. Figure 1 and Figure 2 compare the globally retrieved temperature and specific humidity at 500 hPa with corresponding values from the spatially and temporally collocated European Centre for Medium-Range Weather Forecasts Re-Analysis interim (ERA-interim) data. The temporal and spatial variations of temperature and water vapor are accurately captured using the retrieval algorithm. The cloud properties for each FOV are retrieved simultaneously along with the atmospheric profiles. Figure 3 shows sample Jacobians of temperature and water vapor for one CrIS FOV measurement where a single layer water cloud with an effective visible optical depth of 1.36 is located at 397 hPa. The Jacobians that characterize the differential response of spectral radiance to small perturbations in state variables are based on the accurate radiative transfer relationship which includes the effect of long-wave scattering by clouds. The quality of the spectral fit between observed and retrieved radiances can be quantified by the cost function (Equation (11)). A threshold value for the cost function can be used to label individual retrievals as “converged” or “nonconverged.” Figure 4 shows the difference between the observed and the fitted spectral radiances of one individual CrIS retrieval that is deemed as “converged.” A higher than 80% retrieval convergence rate has been achieved for the CrIS/ATMS measurements at a global scale. The radiometric consistency check of individual retrievals is critically necessary to ensure the quality of the retrieval results to be used for climate studies.

4. Fingerprinting Applications

4.1. Fingerprinting of CrIS Data

We use the fingerprinting method to derive monthly mean temperature and water vapor anomalies from the CrIS spectral radiance anomalies. Our first step is the construction of zonally averaged monthly mean spectral radiance anomalies. Spectral radiances of individual CrIS FOVs in each month are gridded into 5 $\times$ 5 degree zones. The monthly mean spectral radiance within each grid cell is obtained by taking the average of thousands of individual results. The climate trend analysis requires the use of the deseasonalized anomalies that are calculated as the difference between the gridded monthly mean values and the reference values of the corresponding month. Those gridded results can be further spatially averaged to obtain globally or zonally averaged results to facilitate the large-scale climate study.

The PCRTM based single FOV retrieval algorithm is used to process the CrIS/ATMS data from 2016 to 2017. The gridded monthly mean values of spectral radiances and geophysical variables for the year 2017 are used as the reference to derive the anomalies ($\mathit{\Delta }\mathit{d}$ and $\mathit{\Delta }\mathit{\alpha }$ in Equation (1)). The Jacobians from the physical retrieval results of 2017 are aggregated into the 5 $\times$ 5 degree grid to construct the radiative kernels $\mathit{S}$ (in Equation (1)). The radiative kernel for each grid is constructed by merging the spatially and temporally averaged Jacobians of temperature, water vapor, skin temperature, surface emissivity, cloud height, and cloud microphysical properties including effective optical depth and particle size. The nonlinearity term $\mathit{\epsilon }$ (in Equation (1)) for each grid zone is then quantified as $\mathit{\Delta }\mathit{d}-\mathit{S}\mathit{\Delta }\mathit{\alpha }$, respectively. The physical retrieval results are known to have less uncertainty over ocean because the spectral variation of the ocean surface emissivity is less complicated than that of land. As compared with the ascending observations, the descending observations are free of solar scattering so that the retrieval is more straightforward and therefore more accurate. Therefore, we choose to focus on the climate anomaly study using the descending observations over the tropical ocean region in this paper. Figure 5 shows the zonally averaged anomalies of the CrIS spectra over the tropical ocean region. Figure 5 also demonstrates the excellent radiometric consistency between the monthly mean spectral anomalies of CrIS measurements over tropical ocean region and that obtained from the retrieval. Figure 6 illustrates the magnitude of the nonlinearity term, which is nontrivial as compared with the corresponding spectral anomalies shown in Figure 5.

Similar to the physical retrieval, the fingerprinting application is also carried out in the PC domain. The use of PCs helps to filter out the noise in spectral anomalies, reduce the dimension of the fingerprint matrix (${\mathit{S}}^T{\Sigma }^{-1}\mathit{S}$ in Equation (8)), and stabilize the solution. The fingerprint matrix is determined by both the radiative kernel and the nonlinearity term. The fingerprinting solution that follows the generalized least square form (Equation (8)) needs to fit the spectral anomalies with a post fit residual that matches the nonlinearity term in order to find the true temperature and water vapor anomalies. This is achieved by accounting for the spectral correlation feature of the nonlinearity term in the covariance matrix $\Sigma$, i.e., the fingerprint uncertainty. $\Sigma$ is generalized using the ensemble of nonlinearity residuals obtained from the physical retrieval results of two reference years. The ensemble contains thousands of monthly mean spectral residuals derived for all 5 $\times$ 5 degree grid zones. It provides a sufficient coverage for the temporal and spatial variation of the nonlinearity term and therefore a statistically efficient characterization for the fingerprint uncertainty.

The implementation of the optimal detection technique often requires the use of constraints to avoid numerical errors and overfitting. Constraints can be in many forms. A proper constraint such as converting radiance anomaly spectra into the Principal Component (PC) domain and using only significant PCs improves the condition number of the fingerprint matrix while avoiding the over-regularization in the solution. Additionally, we choose to use the statistically obtained a priori constraint that is suggested by Rodgers [35] for the atmospheric sounding inversion method. The solution can be given as

$$\mathrm{\Delta }\mathit{\alpha }={({\mathit{S}}^T{\Sigma }^{-1}\mathit{S}+{\Sigma }_a^{-1})}^{-1}{\mathit{S}}^T{\Sigma }^{-1}\mathrm{\Delta }\mathit{d}$$

(12)

${\Sigma }_{\mathit{a}}$ can be obtained statistically as the covariance matrix of the sampling atmospheric anomalies. The atmospheric anomalies of the gridded samples derived from the retrieval results for our 2016 analysis are used here to calculate ${\Sigma }_{\mathit{a}}$. The uncertainty of the fingerprinting is then determined by the constrained fingerprint matrix,

$$\mathrm{Var}\left(\mathrm{\Delta }\mathit{\alpha }\right)={({\mathit{S}}^T{\Sigma }^{-1}\mathit{S}+{\Sigma }_{\mathit{a}}^{-1})}^{-1}$$

(13)

The atmospheric temperature and water vapor anomalies from the fingerprinting are compared with the physical retrieval results. Figure 7 shows the validation for the tropical ocean region monthly mean temperature anomalies of year 2016 obtained via fingerprinting. Figure 8 compares the corresponding water vapor anomalies. Our fingerprinting scheme is designed to rely on the information provided by the measurements instead of the a priori. Therefore, the magnitude of the fingerprinting uncertainty, i.e., the square root value of the diagonal element in the fingerprint matrix defined by Equation (13), strongly depends on the magnitude of the nonlinearity covariance $\Sigma$. Although the magnitude of the fingerprinting uncertainty (shown as dashed lines in Figure 7 and Figure 8) is large as compared with the magnitude of the anomalies, the properly established covariance matrix $\Sigma$ captures the correlation relationship within the nonlinearity terms and therefore minimizes the errors due to the fingerprinting uncertainty. The systematic difference in the anomalies of temperature and water vapor are well below 0.1 K and 0.1 g/kg at all altitudes, respectively. Figure 9 shows the validation for the monthly mean anomalies of effective cloud optical depth and surface skin temperature. The cloud optical depth anomaly for the twelve months of 2016 ranges from −0.02 to 0.08, while the correlation coefficient between the fingerprinting results and the retrieval results is larger than 0.97. The cloud optical depth anomaly difference between the fingerprinting and the retrieval has a less than 0.001 absolute bias and a 0.006 standard deviation. The bias and the standard deviation values for the surface skin temperature anomaly difference between the fingerprinting and the retrieval are −0.01 K and 0.03 K, with a larger than 0.98 correlation between the two sets of anomalies.

4.2. Fingerprinting of AIRS Data

The PCRTM can be used to simulate various hyperspectral sensors, including both CrIS and AIRS. The simulation for AIRS shares the same forward calculation module with the simulation for CrIS. In the development of the PCRTM, both CrIS and AIRS spectral channel radiances are calculated by convolving high resolution line-by-line monochromatic radiance spectra with the CrIS spectral response function (SRF) and the AIRS SRF, respectively. The SRFs of CrIS and AIRS are well-defined and fixed, therefore, the simulated AIRS and CrIS spectra using the same input parameters are strongly correlated. We can use a linear regression relationship to transform a given CrIS spectrum to a “virtually” collocated AIRS spectrum.

$${\mathit{R}}_{airs}=\mathit{A}{\mathit{R}}_{cris}$$

(14)

The regression matrix can be trained using the PCRTM simulation results for a wide range of atmospheric and cloud conditions. The accuracy of using the regression relationship to transfer CrIS observations into corresponding AIRS observations is validated. Figure 10 shows the prediction accuracy that can be achieved using 10,000 training samples. Figure 11 compares the monthly spectral anomaly of AIRS from 2013 to 2018 with that predicted from the observed CrIS monthly spectral anomaly. The AIRS spectral anomalies are derived from the AIRS Level 1B data [36], following the same procedure of generating temporally and spatially averaged data described in Section 4.1. Despite the spatial and temporal sampling difference, there is a good agreement between the two sets of spectral anomalies. The consistency between CrIS and AIRS measurements facilitates a consistent fingerprinting application on spectra measured by different sensors using common radiative kernels, which provides a means of generating climate continuity products by merging observations from multiple hyperspectral sensors.

We derive temperature and water vapor anomalies from the sixteen years of AIRS observations from January 2003 to December 2018, using the radiative kernels constructed from those used for the CrIS fingerprinting,

$${\mathit{S}}_{airs}=\mathit{A}{\mathit{S}}_{cris}$$

(15)

The nonlinearity covariance matrix is generated in the same way as described in Section 4.1 but using the constructed AIRS kernels of 5 $\times$ 5 degree grid cells and the corresponding AIRS spectral anomalies. Those fingerprinting results are validated using the AIRS Level 3 data [37]. The AIRS Level 3 products have been developed for the analysis of global- or zonal-scale weather and climate studies. AIRS Level 3 products include gridded atmospheric data sets at various temporal resolutions and a 1 $\times$ 1 degree spatial resolution. The monthly mean AIRS Level 3 data is first aggregated into the 5 $\times$ 5 degree grid and then averaged over the tropical ocean region to match the fingerprinting results. Figure 12 compares the time series of temperature anomalies at different altitudes from the fingerprinting results with that from the AIRS Level 3 products. We can see that the month-to-month variations of temperature anomalies at various altitudes from stratosphere to low troposphere are largely captured in the fingerprinting. The differences between the fingerprinting and the retrieval are relatively more obvious in the tropospheric temperature anomalies at 300 hPa, 497 hPa, and 707 hPa during the time period of 2003–2009. Figure 13 shows that the surface skin temperature anomaly generally agrees with that from the AIRS Level 3 products with a less than 0.02 K mean difference. Figure 14 illustrates the comparison between water vapor anomalies obtained from the fingerprinting and those from the AIRS Level 3 products. Although the variation of the water vapor anomalies from the fingerprinting still largely follows the AIRS Level 3 results, there are visible disagreements between the fingerprinting results and the AIRS Level 3 data, especially for the water vapor anomalies of earlier years. Figure 14 also shows that the differences in water vapor anomalies at 300 hPa and 497 hPa are systematically larger from 2003 to 2009 than that after 2009. The water vapor anomalies near surface (at 802 hPa) from the fingerprinting are apparently larger than the AIRS Level 3 results before 2014, although there is a good agreement between the two sets of results after 2014. There are abnormal spikes at June, 2007 in the time series plots of fingerprinting results (the temperature anomaly at 9.5 hPa, the surface skin temperature anomaly, and the water vapor anomaly at 300 hPa). Another obvious spike shows at October, 2003 in the surface skin temperature anomaly plot. Those spikes are associated with the two abnormally large spectral anomalies shown in Figure 15.

Drifts/offsets in the AIRS radiances can introduce errors in both the retrieval and the fingerprinting results. AIRS Level 3 products are generated using the physical retrieval Level 2 algorithm with a cloud-clearing algorithm and a neural-network first guess trained from the numerical weather prediction (NWP) models. The fingerprinting is constructed from the PCRTM based physical retrieval that uses the climatology background as the constraint. The latter constraint is more relaxed so that the fingerprinting results can be more sensitive to the radiance errors. AIRS is a grating spectrometer with known drifts in the AIRS channel centroid frequencies and the Doppler shifts in the AIRS radiances due to the rotation of the Earth seen from the Aqua spacecraft. The AIRS radiance data used in this paper are the AIRS L1C data of version 6 [38]. The frequency shift correction is not applied to this version of AIRS data. Strow et al. [39] characterized the cause of the frequency shifts in AIRS measurements that occurred at different stages of the operation. There are slow detector position drifts in most of the AIRS modules that gradually stopped in 2009. After 2009, there are two kinds of drift patterns: a seasonal variation drift and an orbit phase dependent drift. Both drifts experienced a step change in the magnitude in early 2014, potentially due to the March 2014 AIRS shut down in modules that measure the water bands. Those shift patterns mean that the characteristics of the AIRS spectral error can be systematically different for three different time periods: before 2009, 2009–2014, and 2014–2018. The corresponding impacts on the fingerprinting of temperature and water vapor are therefore different for three time periods. The relatively larger frequency shift errors before the year 2009 lead to larger errors in the fingerprinting of both temperature and water vapor, as is shown in Figure 12 and Figure 14. The water vapor anomaly at 802 hPa from the fingerprinting and that from the AIRS Level 3 results are systematically different before 2014 but in good agreement after 2014, which also coincides with the change of the frequency shift pattern in early 2014. The fingerprinting scheme is constructed using the physical retrieval results of 2016 and 2017 so that the AIRS spectral errors after 2014 are better accounted for in the fingerprinting uncertainty matrix. In general, there is a systematic correlation between the characteristics of the AIRS frequency shift and the fingerprinting errors, which indicates that a significant part of the fingerprinting errors can be caused by the errors in AIRS spectral radiances.

4.3. Comparison Between CrIS and AIRS Fingerprinting during the Overlapping Period

When CrIS and AIRS data records are combined and used to derive long-term climate trends, we need to quantify the difference between CrIS and AIRS results during the overlapping period and evaluate its impact on the climate trend analysis. Figure 16 shows the monthly mean temperature anomalies in time series derived from the zonally averaged CrIS fingerprints and that from the AIRS fingerprints for the period from January 2013 to December 2018. Figure 17 illustrates the agreement and the small differences between the two sets of surface skin temperature anomalies. Figure 18 compares the two sets of humidity anomalies at various altitudes. When using muti-instrument measurements to derive climate data products, the difference between CrIS and AIRS results can be viewed as the measurement introduced uncertainty. The impact of such uncertainty on the long-term climate trend detection needs to be evaluated. By following the methodology used to study the instrument calibration requirement for climate change detection [1,2,3], the impact can be quantified using the trend accuracy uncertainty factor $U_a$, with

$$U_a={(1+{\sigma }_{diff}^2{\tau }_{diff}/{\sigma }_{var}^2{\tau }_{var})}^{1/2}$$

(16)

where ${\mathsf{\sigma }}_{\mathrm{diff}}$ is the standard deviation value of the CrIS-AIRS difference (lag-0 uncertainty) and ${\mathsf{\tau }}_{\mathrm{diff}}$ is the corresponding autocorrelation time. ${\mathsf{\sigma }}_{\mathrm{var}}$ and ${\mathsf{\tau }}_{\mathrm{var}}$ are the standard deviation value and the autocorrelation time associated with natural variability, respectively. If the CrIS and AIRS results are perfectly consistent, the trend uncertainty will be solely determined by the natural variability variance factor ${\mathsf{\sigma }}_{\mathrm{var}}^2{\mathsf{\tau }}_{\mathrm{var}}$ and the length of the detection time. ${\mathrm{U}}_{\mathrm{a}}$ here defines the ratio of trend uncertainty introduced by both the CrIS-AIRS measurement difference and the natural variability to that only introduced by the natural variability. For example, $U_a=1.2$ means that the CrIS-AIRS measurement inconsistency increases the trend uncertainty by 20% as compared with the trend accuracy established from perfect measurements. By using the AIRS Level 3 data products (shown in Section 4.2) as the reference to estimate ${\sigma }_{var}$ and ${\tau }_{var}$, we can evaluate the impact of CrIS-AIRS measurement uncertainty on the temperature and water vapor trends at different altitudes. The results are listed in

Table 2. CrIS-AIRS fingerprinting difference and the impact on the climate trend detection.

Parameters at Different Pressure Levels	${\mathit{\sigma }}_{\mathit{d}\mathit{i}\mathit{f}\mathit{f}}$ k = 1	${\mathit{\tau }}_{\mathit{d}\mathit{i}\mathit{f}\mathit{f}}$ (Month)	${\mathit{\sigma }}_{\mathit{v}\mathit{a}\mathit{r}}$ k = 1	${\mathit{\tau }}_{\mathit{v}\mathit{a}\mathit{r}}$ (Month)	${\mathit{U}}_{\mathit{a}}$
Temperature @ 9.5 hPa	0.22 K	3.87	0.82 K	1.98	1.07
Temperature @ 300hPa	0.11 K	5.76	0.46 K	7.24	1.02
Temperature @ 497hPa	0.21 K	35.39	0.33 K	4.46	2.06
Temperature @ 707hPa	0.19 K	24.88	0.29 K	5.55	1.72
Temperature @ 802hPa	0.21 K	24.04	0.23 K	7.04	1.94
Surface Skin Temp.	0.15 K	4.26	0.24 K	9.18	1.09
H2O @ 300hPa	0.003 (g/kg)	2.80	0.012 (g/kg)	4.32	1.02
H2O @ 497hPa	0.064 (g/kg)	6.52	0.088 (g/kg)	4.48	1.33
H2O @ 707hPa	0.167 (g/kg)	6.71	0.183 (g/kg)	5.72	1.40
H2O @ 802hPa	0.157 (g/kg)	6.19	0.185 (g/kg)	5.45	1.35

. The parameters ${\tau }_{diff}$ and ${\tau }_{var}$ are obtained by fitting the CrIS-AIRS differences and the AIRS anomaly time-series data to corresponding autoregressive and moving average models using the approach in [28]. From Table 2, we can see that measurement uncertainty factor $U_a$ for atmospheric temperature and water trends detection is the lowest at 300 hPa. This is in line with the excellent agreement in the temperature and water vapor anomalies at 300 hPa between CrIS and AIRS shown in Figure 16 and Figure 18. Such a good agreement between CrIS and AIRS results suggests the feasibility of building continuous long-term data records of temperature and water vapor anomalies at 300 hPa through the fusion of data from multiple hyperspectral sensors. The impact of CrIS-AIRS anomaly differences on both the stratospheric temperature and the surface skin temperature trends is relatively insignificant. However, the impact becomes larger in the lower tropospheric region and can increase the trend uncertainty by more than 100%. This also illustrates the fact that the fingerprinting errors are larger in the mid-to-low altitude region because of the limited sensitivity of the IR measurements to the radiative contribution from below clouds.

5. Conclusions and Future Work

Climate spectral fingerprinting has been demonstrated as a promising approach to efficiently carry out climate trend and anomaly studies using the OLR spectral measurements. In this paper, we introduce a novel fingerprinting scheme that can be used to derive atmospheric anomalies from real hyperspectral sounder data. We demonstrate for the first time the use of the fingerprinting method to derive monthly mean anomalies of temperature and water vapor profiles over the tropical ocean regions from the CrIS and AIRS observations under all-sky conditions. The fingerprinting scheme follows a generalized least squares approach with the radiative kernels and the fingerprinting uncertainty established from two years of combined CrIS+ATMS retrieval results. It is successfully used to obtain temperature and water vapor anomalies from CrIS spectral anomalies of 2013–2018. The fingerprinting scheme can be easily transformed into the AIRS spectral domain using the regression relationship constructed from the PCRTM-based simulations and subsequently applied to AIRS spectra. We have derived temperature and water vapor anomalies from the sixteen-year-long data record of AIRS, using the fingerprinting relationship constructed from the CrIS retrieval results. Those results are validated using the independent AIRS Level 3 data. The highlights of the fingerprinting algorithm are listed in the following:

(1)

The fingerprinting scheme is constructed from the PCRTM physical retrieval results to ensure the radiometric consistency between the simulation and the observations, which cannot be satisfied by using results from climate models or numerical reanalysis or by using cloud-clearing products.

(2)

The capability of providing rigorous radiometric fitting to observations under all-sky conditions depends on the cloud scattering simulation capability built in the PCRTM. As a comparison, operational hyperspectral sounder algorithms avoid cloud scattering simulations by using the “cloud-cleared” radiance spectra to retrieve temperature and water vapor profiles. The radiance closure needed for climate studies cannot be established using the current AIRS and CrIS operational data products due to the lack of cloud information from the corresponding radiative transfer simulations.

(3)

The fingerprinting scheme used for AIRS fingerprinting is consistent with that used for CrIS fingerprinting. Therefore, the discrepancy between AIRS and CrIS fingerprinting results introduced by the algorithm itself is minimized.

(4)

Through its consistency with the physical retrieval that uses the high vertical resolution profiling capability of hyperspectral sounders, the optimal fingerprinting also provides the vertical profiling of atmospheric anomalies that is useful for climate feedback studies.

(5)

The radiometrically consistent fingerprinting facilitates the characterization for the impact of instrument related measurement errors on the fingerprinting results. The difference between the fingerprinting results from the overlapping observations of CrIS and AIRS can be used to evaluate the impact of temporal-spatial sampling and instrument calibration related differences on the determination of climate trends using the combined data set.

Future work will focus on the improvement of the fingerprinting accuracy and the extension of the fingerprinting application for a global scale data analysis. We notice that a significant part of AIRS fingerprinting errors can be attributed to the AIRS spectral error due to the shifts in AIRS channel centroid frequencies. The frequency shift error can be reduced by characterizing the contributing factors such as the detector position drifts and the Doppler drifts between the Earth and the Aqua satellite [39]. We expect better temperature and water vapor fingerprinting results by using the updated version of AIRS L1C data with frequency shift errors being corrected. The fingerprinting errors in the atmospheric anomalies at low altitudes are larger than those in the high-altitude anomalies, due to the limited sensitivity of IR measurements to atmospheric profiles below clouds. We plan to carry out the combined IR+MW fingerprinting study in the future to improve the fingerprinting accuracy by using the MW signals to enhance the detection for atmospheric anomalies below clouds. The fingerprinting study using the hyperspectral sounder observations over the tropical ocean regions has been demonstrated and validated in this paper. The same fingerprinting methodology can be easily extended for the climate anomaly and trend study over other climate zones including midlatitude and polar regions. The climate trends analysis for smaller gridded areas at various latitude-longitude regions can also be carried out. We can also use the method to derive atmospheric anomalies from IASI data. The comparison between the IASI results and those from CrIS and/or AIRS can be used to characterize the impact of the local sampling time differences on the observation of climate trends.