Review on Analysis Methods Enabled by Hyperspectral Imaging for Cultural Relic Conservation

Abstract

In this review, the conservation methods for various types of cultural relics enabled by hyperspectral imaging are summarized, and the hyperspectral cameras and techniques utilized in the process from data acquisition to analyzation are introduced. Hyperspectral imaging is characterized by non-contact detection, broadband, and high resolution, which are of great significance to the non-destructive investigation of cultural relics. However, owing to the wide variety of cultural relics, the utilized equipment and methods vary greatly in the investigations of the associated conservation. Previous studies generally select a single type of cultural relic for conservation. That is, seldom study has focused on the application of hyperspectral techniques to generalized conservation methods that are simultaneously suitable for different types of cultural relics. Hence, some widely used hyperspectral cameras and imaging systems are introduced first. Subsequently, according to the previous investigations, the methods used for image acquisition, image correction, and data dimensionality reduction in hyperspectral techniques are described. Thirdly, a summary of methods in cultural relic conservation based on hyperspectral techniques is presented, which involves pigments, grottoes and murals, and painting and calligraphy. Later, some challenges and potential development prospects in hyperspectral-based methods are discussed for future study. Finally, the conclusions are given.

1. Introduction

There is a great number of cultural relics around the world produced through its long history. These relics are an important cultural heritage that carries the history of the progress of people in the world and invaluable humanistic knowledge; thus, their conservation is essential and has attracted increasing attention in recent years. In the early years of heritage exploration, a manual approach with expert experience was the mainstream solution. However, owing to the limitations of our knowledge, it is difficult to monitor and preserve severely damaged heritage well, and human intervention is even more likely to cause irreversible damage to the heritage. Therefore, various chemical and physical analytical methods have been gradually introduced into the study of heritage conservation.

Spectral analysis techniques including X-ray fluorescence and laser Raman spectroscopy have been successfully used in cultural relic conservation for analyzing material elements, identifying pigment composition [1], deconstructing rock formation [2], and other functions. Raman spectroscopy can provide highly detailed structural information of molecules, but its small light spot size makes it unsuitable for analyzing cultural relics with large sizes. Later, digital image analysis was introduced to fully describe the spatial information of cultural relics. Nevertheless, digital images have strict requirements on the environmental lighting conditions and are limited to the visible light range, thus resulting in notable information loss along the near- and mid-infrared wavelength range. The multispectral method uses both visible and invisible lights to describe the spectrum of cultural relics and identify its pigment composition [3]. However, multispectral analysis is usually characterized by wide spectral intervals, which may drown small differences in sensitive bands and eventually lose some detailed information. With the development of optoelectronics, hyperspectral imaging, first developed in the 1980s by the Jet Propulsion Laboratory and previously used in medical diagnosis and food safety, was first utilized for the pigment identification of cultural relics in 2003 [4]. Hyperspectral imaging that can extend the spectral range to the mid-infrared band with much smaller spectral intervals has been utilized for the information extraction of paintings and calligraphy [5,6,7,8,9,10,11], pigment identification [12], and the paint loss disease of grottoes and murals [13,14,15].

In [16], several conventional imaging techniques for cultural relic conservation are summarized, including X-ray, multispectral, and hyperspectral imaging, as well as the corresponding devices and their parameters. In [17], the achievements of hyperspectral imaging in painted artefacts for implied information mining and pigment analysis are concluded. Similarly, the hyperspectral method is also used for rock and mineral identification. Different hyperspectral remote sensing methods for rock and mineral identification, including their advantages, disadvantages, and applicable scenarios, are summarized in [18]. In this paper, the detailed processes of cultural relic conservation based on hyperspectral imaging, from data acquisition to data application, are introduced first. Subsequently, the various types methods of cultural relic conservation using hyperspectral imaging are discussed in detail. Finally, the current insufficiencies and several potential improvements of cultural relic conservation enabled by hyperspectral imaging are presented.

2. Features of Hyperspectral Imaging

The past decades have witnessed the rapid development of spectral analysis, and currently there are different instruments covering various wavelengths. Several typical spectrometers and their major parameters are listed in

Table 1. Parameters of several representative spectrometers.

Spectrometer	Spectral Range	Resolution	Image	Spectral Information
Raman Spectrometer	2.5–25 μm	<15 cm−1	Chemical images with molecular information	Raman scattering of molecules
Short-wavelength Infrared Spectrometer	1000–2500 nm	<12 nm	Optical images	Reflectance of each pixel
Hyperspectral Spectrometer	400–2500 nm	<10 nm	Optical images	Reflectance of each pixel

. The hyperspectral imager is an optical instrument that can acquire spectral information like reflection, scattering, or radiation from an object at different wavelengths and transform it into high-resolution images with 100–400 spectral channels. As shown in Table 1, compared to the Raman and short-wavelength infrared spectrometers, hyperspectral imagers exhibit a high resolution of less than 10 nm, and generally operate in a spectral range of 400–2500 nm [19], covering both the visible and near-infrared light ranges. As for cultural relic conservation, hyperspectral images are characterized by non-contact detection, are non-destructive, and have a wide wavelength range. The spectrometers have the feature of “combining image with spectrum acquisition”, which can preserve the spatial characteristics and spectral information of cultural relics; hence, multi-angle methods can be further used with hyperspectral images.

Hyperspectral images are different from digital images in that they include both the spectral information and the spatial information. Taking the spatial information of the image as two-dimensional (x, y) and the spectrum as the third dimensional information, each pixel of the image contains detailed information with hundreds of bands. The spectral information is generally expressed by reflectance [20]. As shown in Figure 1, for a certain point of the hyperspectral image, its reflectance curve versus the wavelength can be easily obtained. For the same matter, its reflectance curve is fixed and directly represents the rule of reflectance variation of the object along with the wavelength variation, which is mainly determined by the physical properties of the matter and will not be influenced by environmental changes. There are certain conditions that can cause subtle differences in the reflectance spectrum of the same matter. For instance, the long-term placement of the object can cause red shift, that is, the entire spectral curve shifting towards the red light. In general, there are obvious individualized differences in the spectral profiles of matters, which are the basis of hyperspectral cultural relic conservation. In fact, the high resolution and large wavelength range of hyperspectral imaging can significantly enhance the efficiency of data processing and analysis.

The compact airborne spectrographic imager (CASI) based on hyperspectral imaging was developed in 1989 [21]. In 2000, National Aeronautics and Space Administration launched the Earth Observation 1 (EO-1) satellite with the Hyperion imaging spectrometer for spaceborne hyperspectral remote sensing (HRS) application [22]. The EO-1 satellite has been operating successfully for over 20 years, providing a large number of HRS images. The Shenzhen-3 China Moderate Resolution Imaging Spectrometer is a spaceborne hyperspectral imager launched in 2002. Since then, a series of spectrometers, including the visible and near-infrared imaging spectrometer used in the Yutu lunar rover, have been developed [23].

In the past two decades, a variety of hyperspectral imaging systems have emerged to provide powerful capability for cultural relic conservation. Dvoptic, Headwall, and XIMEA have developed hyperspectral scanning systems to study paintings [24,25,26]. The VSC6000 system commercialized by Foster+Freeman has been widely used for handwriting analysis [27]. The National Gallery of Art reflectance hyperspectral systems [9] extend the spectral range to 2500 nm with a 2.8 nm spectral sampling wavelength, which can support the mapping of paint binders and improving pigment identification. The Palace Museum, together with the Remote Sensing Institute of the Chinese Academy of Sciences, has developed a hyperspectral automatic scanning and stitching system specifically for the analysis of paintings and calligraphy [28]. Additionally, compact and portable hyperspectral cameras have been widely used in cultural relic conservation.

Table 2. Parameters of commonly used hyperspectral cameras.

Hyperspectral Camera	Brand	Spectral Range	Resolution	Analysis Objects
VNIR A-Series hyperspectral imager	Headwall	380–1000 nm	3 nm	Murals
VNIR 400H hyperspectral camera	Themis Vision Systems	400–1000 nm	2.8 nm	Murals, paintings
Pro-V10 hyperspectral camera	Dualix	400–1000 nm	3.5 nm	Murals
SOC710 hyperspectral camera	Surface Optics	400–1000 nm	2.4 nm	Murals, crockery
SVC1024i portable spectrometer	Spectra Vista	340–2510 nm	≦2.8 nm, 350–1000 nm ≦8.0 nm, 1000–1900 nm ≦6.0 nm, 1900–2500 nm	Pigments, porcelains
GaiaField imaging camera	Dualix	385–1032 nm	3.5 nm	Pigments
FieldSpec3 portable spectrometer	Malvern Panalytical	350–2500 nm	3 nm, 350–1000 nm 10 nm, 1000–2500 nm	Pigments
T-FPS2500 hyperspectral camera	Themis Vision Systems	400–2500 nm	6.3 nm	Murals, paintings
NUVNIR-350 pushbroom hyperspectral imager	Themis Vision Systems	350–1000 nm	1.5 nm	Paintings

sketches four main parameters of some commonly used hyperspectral cameras. The SVC1024i portable geospectrometer and Dualix GaiaField imaging camera are generally used in pigment identification. The visible and near-infrared (VNIR) 400H hyperspectral camera from Themis Vision Systems is capable of extracting and identifying faded text [29], and has been used to study tomb murals [30]. Furthermore, the Headwall short-wave near-infrared unmanned airborne hyperspectral imager is suitable for extreme environmental conditions like cliffs and crags. In [31], a novel hyperspectral camera with effective pigment mapping was developed for painting imaging, which shows that the Fourier transform hyperspectral imaging is a promising technique for cultural relic conservation.

3. Hyperspectral Image Analysis

To obtain spectral data with higher reproduction and eliminate the influence of environmental noise, hyperspectral images should be processed properly at each stage: hyperspectral image acquisition, spectral image correction, and data dimensionality reduction. The methods used in these stages are described in detail below.

3.1. Hyperspectral Image Acquisition

Hyperspectral image acquisition can be carried out by using different kinds of hyperspectral imagers, such as close-up shooting cameras, airborne cameras, and satellite-based cameras. The spaceborne and airborne spectrometers are able to receive ground reflection information in different spectral bands including visible, near-infrared, mid-infrared, and others. Under the geographical conditions of dry surface soil, archaeological remains can form more obvious signs in hyperspectral images; in particular, their mid-infrared band has a better reflection effect on the subsurface remains and can restore the general layout. As is known, the flight altitude of aircraft (6–15 km) is much lower than that of satellites (400–1000 km), thus the spatial resolution of airborne hyperspectral imaging is generally much higher than that of the satellite platforms. Hence, in archaeological research, airborne spectrometers are more attractive than spaceborne ones [32]. Therefore, airborne hyperspectral remote sensing has been utilized to detect the overall layout of the underground part of Qin Shihuang’s Mausoleum [33].

In fact, there is a large number of small objects in cultural relic conservation. For the murals, paintings, and ceramics, the close-range photography-based approach exhibits notable advantages in spatial resolution and local details expression. To retain as much detail as possible in the original images, the effects of lighting and other environmental factors must be eliminated. In the study of Han Xiu’s Tomb murals, Headwall’s VNIR A-Series hyperspectral imager with a spectral range of 380–1000 nm was used to perform hyperspectral imaging. Specifically, the spectrometer was installed 2–3 m away from the mural, with a tungsten lamp with a large field of view installed next to it. As the light source, the tungsten lamp could translate along the track with the spectrometer for imaging, thus overcoming the uneven illumination resulted from the fixed light source condition [34]. Similarly, by using a Headwall imaging spectrometer with a spectral range of 400–2500 nm, the “Tablet of Buddha written by Cixi” in the Forbidden City has been studied with the hyperspectral imaging method [28]. The hyperspectral images were acquired by placing the calligraphy flat on a table with a halogen lamp as the light source, setting the spectrometer parallel to the calligraphy, and rotating the scanning mirror to complete the scan imaging of the calligraphy. In fact, the tungsten and halogen lamps both contain abundant near-infrared wavelength light, which has a better color reproduction effect on the artifacts. They also have a variety of light forms like direct radiation, scattering, and diffusion, which are able to achieve a uniform illumination of the surface of cultural relics. As shown in Figure 2, a tungsten iodide lamp with similar properties has been widely used in hyperspectral artworks acquisition and also meets the requirements for light illumination [35].

3.2. Hyperspectral Image Correction Methods

Since the image acquisition is subject to interference from various environmental factors in different scenarios, the acquired hyperspectral images cannot be directly used for spectral analysis. Generally, images acquired indoors or by remote sensing are influenced by the non-uniform illumination of light sources, thereby requiring reflectance correction. Moreover, hyperspectral images of surface relics are greatly influenced by topography, which results in using a remote sensing topography correction method to exclude the negative effects of topographic factors. Therefore, before processing the hyperspectral image data, proper correction with fully consideration of the characteristics of hyperspectral acquisition can effectively reduce the difficulty of data analysis and eventually improve its efficiency.

Subsequently, as the most commonly used correction methods of spectral reflectance, the standard whiteboard and dark current correction is suitable for close-range imaging and remote sensing imaging. To eliminate the error caused by non-uniform illumination, point-to-point reflectance correction for the spectral images is carried out using a standard diffuse reflection board. Then, the light source of the camera and the halogen lamps are turned off, and the spectral images are captured under dark current conditions to exclude the noise from the instruments. Therefore, the spectral reflectance after processing can be expressed as:

$$f=\frac{f_{\mathrm{c}}-f_0}{f_{\mathrm{b}}-f_0}$$

(1)

Here, f_c, f_b, f₀ denote the spectral data, whiteboard data, and dark current data of the acquired images, respectively.

As a hyperspectral imager reflectance correction algorithm, the average method of standard whiteboard and dark current correction has been proposed to reduce the phenomenon of “high-drifting reflectance” in the conventional algorithm [36]. Afterwards, a batch processing approach based on the Environment for Visualizing Images (ENVI) software is developed to improve the efficiency of data processing. In the average method, let the size of the image data’s original value be K bands of m rows and n columns. Then, the row-average value of the K-th band of the dark current data Ba_(n,k) is

$$B{a}_{(n,k)}=\frac{{\displaystyle \sum _{i=1}^m{B}_{m\times n\times k}}}{m}$$

(2)

where B_m×n×k denotes the dark current data of the m-th row and n-th column of the K-th band. Similarly, the row-average value of the K-th band of the whiteboard data can be calculated as

$$W{a}_{(n,k)}=\frac{{\displaystyle \sum _{i=1}^m{W}_{m\times n\times k}}}{m}$$

(3)

where W_m×n×k denotes the whiteboard data in the m-th row and n-th column of the K-th band. By substituting the aforementioned data into the correction formula of the average method, the reflectance ρ_DN can be calculated as

$${\rho }_{\mathrm{DN}}=\frac{D{N}_{(k,j)}-B{a}_{(n,k)}}{W{a}_{(n,k)}-B{a}_{(n,k)}}$$

(4)

where DN_(k,j) is the data in the j-th row of the K-th band. The average method does not represent the overall data quality well due to the noise in the data. The improved average method performs the overall reflectance inversion on the original image data ρ_DN(k) as

$${\rho }_{\mathrm{DN}(k)}=\frac{{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^{n}D{N}_{(m,n)}-}{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n{B}_{(m,n)}}}}}{{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n{W}_{(m,n)}-{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n{B}_{(m,n)}}}}}}$$

(5)

The calculation process is implemented in batch using ENVI, which reduces the workload in multiple image processing.

Under the imaging conditions with complex topography, eliminating the influence from topography is also significant to improve the quality of hyperspectral imaging. Several models have been proposed for the topography correction of hyperspectral remote sensing [37], and the Minnaert + SCS correction model was found to be optimal through the comparison of parameter evaluation. This model can set up an environmental correction method under non-Lambertian conditions. It introduces the solar zenith angle parameter, which has a good effect on the difference of solar radiation between the shaded and sunny slopes of mountains. Another spectral correction model based on the Kubelka–Munk theory was proposed to remove the influence from moisture in the hyperspectral soil composition analysis [38], which has provided a good alternative for excluding the moisture interference with soil reflectance. In practical cultural relic conservation, appropriate image correction methods should be used for different acquisition scenarios to provide reliable data for the subsequent processing. For ground-based hyperspectral imaging techniques, the standard whiteboard method is the most commonly used approach for image correction in cultural relic conservation. Additionally, some techniques do not take the effect of light sources on reflectance correction into account. This is mainly due to the fact that the small size of the object in close-range imaging makes it easier to form ideal diffuse reflectance conditions, thereby reducing the influence from the non-uniform light source. However, in conservation by remote sensing, cultural relics like grottoes and tombs are usually large and immovable, which requires the use of a large-scale artificial light source for hyperspectral imaging, and consequently makes it difficult to create ideal diffuse reflection conditions. Generally, in accordance with the spatial distribution of the light source, it can be divided into three different types, namely, the double-point light source, the multi-point light source, and the linear light source. In practical applications, it has been found that the light intensity of the double-point light source decreases with the increase of the distance from the center of the light source. The variation of light intensity greatly affects the accuracy of reflectance correction. Nevertheless, most conventional reflectance correction methods have not taken this factor into consideration, which causes a certain error in reflectance correction. Therefore, improving the accuracy of conventional reflectance correction methods is urgently demanded.

3.3. Hyperspectral Data Dimensionality Reduction

With the feature of “uniting image and spectrum”, hyperspectral images can provide spectral information in tens to hundreds of bands for each pixel, which greatly increases the amount of data, but also brings in information redundancy. Hence, it is essential to reduce the data dimensionality of the acquired hyperspectral images, so as to streamline the information redundancy while retaining the information of important feature-sensitive bands of cultural relics. Appropriate dimensionality reduction can squeeze the data volume and improve the accuracy of data analysis. Currently, there are two main methods for the dimensionality reduction of hyperspectral images data, i.e., feature selection and feature extraction [39]. As shown in Figure 3, feature extraction includes the processes of information synthesis, feature enhancement, and spectrum downscaling for each spectral band, which converts the original spectral space into a low-dimensional space through a specific mapping rule. There are various types of spectral feature extraction depending on the mapping rules.

As a basic feature extraction method for hyperspectral data, principal component analysis (PCA) is based on an orthogonal linear transformation, allowing to map the data into a new coordinate space in accordance with the distribution of information components. The variance of the data is used to evaluate the content of information. Therefore, the variance of the corresponding principle components can be expressed as

$$Var({\mathrm{z}}_i)=a_i^{\mathrm{T}}{\displaystyle \sum a_i} , i=1,2,\dots p$$

(6)

Here, a_i is the i-th transform vector, ∑ is the covariance matrix of the original data, and z_i is the i-th principal component of the low-dimensional space.

Thereafter, the selective PCA method was proposed in [40], which can accurately predict the object’s feature information and map it to a component. Unfortunately, this method is less effective for the non-obvious spectral features. A PCA-based spectral feature extraction method was proposed in [41], which has exhibited good performance in the hierarchical classification of urban manmade objects. The limitation of the PCA method is that its information-amount-based mapping process cannot deal with the noise very well; hence, the result will be significantly deviated as the images contain complex noise components.

Minimum noise fraction (MNF) is another popular hyperspectral feature extraction method. Unlike the PCA method, the MNF method ranks the components according to the image quality, which can be measured by the noise fraction. Then, the noise fraction is expressed as

$$NF=\frac{a^{\mathrm{T}}{S}_{\mathrm{N}}a}{a^{\mathrm{T}}Sa}$$

(7)

where a is the transformation matrix, S_N is the covariance matrix of the noise, and S is the covariance matrix of the original data.

The MNF method effectively solves the problem arising in PCA that the image quality decreases as the noise distribution is non-uniform. However, the conventional MNF transform estimates the covariance matrix of the noise in the spatial domain, where it is difficult to eliminate the influence of errors. Therefore, an optimized maximum noise fraction (OMNF) method estimating the covariance matrix of the noise in the spectral domain is proposed in [42]. It can be easily obtained from Figure 4 that the accuracy of data classification with the OMNF method is better than that of the conventional MNF one. Recently, the MNF transform has been utilized to extract blurred seals in calligraphy and painting [43]. However, the seals in the hyperspectral images at 300–1000 nm were still blurred with direct MNF transform. To improve the clarity, the hyperspectral image at 350–1000 nm was firstly segmented into 400–600 nm and 500–850 nm bands, and then the MNF transform was performed separately. It has been shown that the images of both the 400–600 and 500–850 nm bands became much clearer compared to those of the 350–1000 nm band, which indicates that selecting the appropriate band before the MNF transform can be helpful in image processing.

As another important method for dimensionality reduction, spectral feature selection requires selecting a simplified subset of the spectral space that contains the main feature of a specific object, as shown in Figure 5. In multi-target recognition, the feature space should be able to represent the features of the corresponding object to the maximum extent possible, so as to distinguish it from others. Feature selection aims to select a subset of bands that retains as much original information as possible and improves image differentiability; thus, it can be regarded as a combinatorial optimization problem, with the criterion for evaluating band combinations being the evaluation function. As the most important constituent of the feature selection algorithm, the evaluation function directly determines the quality of band selection.

Depending on whether a priori information of the feature spectrum is needed in establishing the evaluation function, the feature selection algorithms can be classified into two categories, namely, supervised and unsupervised methods. Unsupervised methods are suitable for application scenarios lacking a priori information on feature classes, and generally use the amount of information as the evaluation criterion for band selection. Generally, parameters of the evaluation function in unsupervised methods include the information entropy, the first-order spectral derivative, the second-order spectral derivative, the number of bands, and others [44,45]. An unsupervised band selection method for hyperspectral remote sensing images based on the spatial spectral genetic algorithm was proposed in [46]. This method can effectively select a subset of bands with low redundancy and high resolution, eventually weakening the “curse of dimensionality”. As feature class information is available, supervised band selection algorithms can effectively retain the bands with class information, thereby improving the classification accuracy of hyperspectral images. Supervised algorithms generally use separability [47] and minimum estimated abundance covariance [48] as the evaluation criteria.

4. Cultural Relic Conservation Enabled by Hyperspectral Imaging

Hyperspectral imaging can provide both spatial image and spectral information of the measured object, then identify and classify the object according to the spectral characteristics of different matters. Therefore, cultural relic conservation enabled by hyperspectral imaging has been widely used to identify and analyze pigments, murals, ancient paintings, and calligraphy. In this section, the application of hyperspectral imaging to different cultural relics and the related methods are discussed in detail.

4.1. Pigment Identification

After years of aging and damage, most surviving cultural relics require the restoration of their pigments. As an important basis for restoration, pigment identification can determine the composition of pigments of cultural relics through analysis. Conventional pigment identification mainly relies on chemical analysis, which is time-consuming and labor-intensive, and its destructive sampling may cause irreversible damage to the cultural relics. For instance, using X-ray irradiation can reveal the elements and chemical compositions of the pigment, but acquiring the pigment samples immediately damages the cultural relics [49,50]. Hence, identifying the compositions of pigments without damage is increasingly demanded in cultural relic conservation. Hyperspectral imaging is one of the safest and most reliable detection techniques. Its feature of “uniting image and spectrum” makes it applicable to the acquisition of cultural relic images, which has great advantages in pigment identification. Therefore, hyperspectral imaging has been gradually used for pigment identification in cultural relic conservation.

In hyperspectral pigment identification, spectral curves of mineral pigments are usually compared with the spectral curves of standard pigment for spectral characteristic matching, so as to identify the compositions of mineral pigments [51]. In the study of mural pigments in the Jokhang Temple in Lhasa [52], images of both the standard and mural pigments are collected by a hyperspectral camera, then the reflection spectral curves are obtained using the endmember extraction function in the ENVI software. Figure 6 sketches the wavelength of the characteristic peaks and the corresponding reflectance. It can be seen that pigments of the same type have closely similar characteristic peaks in a specific wavelength. Some pigments lack an obvious characteristic peak in their spectral curves. For instance, indigo has two peaks at ~840 nm and ~960 nm, respectively, with non-obvious changes in reflectance around these two wavelengths, as shown in Figure 7. Fortunately, it can be obtained that the reflectance curve of indigo increases most rapidly around ~730 nm and ~910 nm, which can be the peaks in the first-order derivative; hence, the first-order derivative of the reflectance curve can be used to identify the compositions.

Table 3. Locations of the characteristic peaks of the reflectance and its first-order derivative curves of various standard pigments.

Pigment	Color	Hyperspectral Standard Pigments Test Results
		Characteristic Peak Location of the Reflectance Curve (nm)	Characteristic Peak Location of First-Order Derivative of the Reflectance Curve (nm)
Azurite	Blue	~450/~460	/
Ultramarine blue	Blue	~450	~720
Indigo	Blue	/	~730/~910
Malachite	Green	~530	/
Emerald green	Green	~495	/
Gamboge	Yellow	/	~535
Orpiment	Yellow	/	~495
Lead chrome yellow	Yellow	/	~530
Realgar	Orange	/	~565
Plumbum rubrum	Orange	/	~450/~620
Lead tetroxide	Orange	/	~575/~720
Shellac	Red	/	~600/~735
Cinnabar	Red	/	~615
Ferric oxide	Red	/	~590/~710
Rouge	Red	/	~670/~685/~705

summarizes the locations of the characteristic peaks and first-order derivative peaks of the reflectance curves of various standard pigments, which are consistent with the acquired mural pigments. Finally, X-ray fluorescence spectroscopy is used to detect the chromogenic elements of the pigments, and the accuracy of the hyperspectral pigment identification is verified by the detection.

Previous investigations tended to identify mineral pigments by matching their spectral curves with the spectral libraries of standard pigments [53], such as the INFRA-ART open access spectral library from Europe and the United States Geological Survey standard spectral library [16]. However, some of the mineral pigments have not been collected in the standard spectral libraries and their spectral curves are unknown. To identify pigments rapidly without using the standard pigment libraries, the hyper-spectrum of red pigments of cultural relics is discussed in [54], and ten red mineral pigments are selected for the study. The normalized spectral indexes of the ten pigments are calculated based on the characteristics of the spectral curves, which can build a distinction model for each red pigment. This method performs well with high recognition accuracy, speeding up the extraction of pigment information by simple exponential operations, which shows practical significance for the rapid and accurate identification of pigments in cultural relics without the standard pigment library. However, it has only investigated a number of red pigments; hence, it is of interest to explore whether the method is also suitable for other pigments. It is worth noting that the same mineral pigment in different physical forms may exhibit different spectral reflectance properties. In [55], five commonly used mineral pigments on thangkas were selected for the study. By analyzing the spectral characteristics of the same mineral pigment in the forms of powder, blended bone gum, and fabric color card, it is found that the overall reflectance of the blended bone gum pigment decreases, while the spectra of the powder and the fabric color card were much closer, with a difference of only around 1920 nm. Hence, the mineral pigment powder can be directly used to analyze thangka pigments. In addition, there are spectral characterization studies based on composite pigments [56,57]. In a word, hyperspectral imaging is widely used in pigment identification, and will be further utilized in a large number of application scenarios.

4.2. Grottoes and Murals

Grottoes and murals contain rich historical knowledge and ancient religious culture. However, owing to years of aging, with the involvement of natural and human factors, the conservation of grottoes and murals has been less optimistic. Generally, diseases of grottoes and murals include pigment layer peeling, salt efflorescence, soot damage, cavitation, cracks, mildew, and others. Hence, using appropriate techniques to detect, mark, and repair the damaged parts of grottoes and murals is highly demanded in cultural relic conservation. The weathering of the surface of grottoes and murals is an important indicator of the health of the rock. However, visual observation cannot accurately describe the degree of weathering, and artificially marking the disease requires a large number of professional staff and a great deal of time [58]. To improve the efficiency of disease investigations, a series of methods to characterize the degree of rock weathering, such as the spreading resistance method [59] and ultrasonic transmission wave method [60], have been proposed. Unfortunately, the above methods also cause damage to the grottoes and murals.

Image classification is applied in state-of-the-art disease labeling in grottoes. In a study of the weathering status of grottoes [61], the spectral data of the weathered sandstone are measured by hyperspectral imaging, and the average reflectance is observed in the sensitive band of 900–920 nm. In the statistical results, weathering areas have a higher reflectance than intact areas. If the average reflectance is higher or lower than 0.58, the area is considered as a strong or slight weathering area, respectively. The technical diagram is shown in Figure 8. The spectral data of different areas also reveal that parts restored with modern cement do not have characteristic peaks in the sensitive band; thus, hyperspectral imaging can also be used to identify the artificially restored parts. Furthermore, the types of minerals on the surface of salt-weathered and dust-weathered areas are different, thereby requiring precise criteria for evaluating the degree of weathering. An intelligent method for evaluating the surface weathering of grottoes based on the random forest algorithm has been proposed in [62], as shown in Figure 9. Firstly, spectral data of complex weathering areas are acquired by using the multispectral technique. Then, the grottoes are classified into four classes, namely, strong salt weathering, weak salt weathering, micro salt weathering, and dust accumulation, by using the random forest algorithm. It has been shown that the prediction accuracy can reach 98.49%, which greatly improves the accuracy of weathered areas recognition.

Murals contains more color information than grottoes, which means that there is more hidden information for mining. In [14], the PCA method is used to analyze a mural in Pompeii, which shows the potential of hyperspectral imaging to highlight hidden details. Original hyperspectral images are firstly compressed into 10 new images by performing PCA, then the false color images are obtained by combining the 1st, 2nd, and 3rd PCA images in the RGB color channels. Specifically, the false color images can highlight the ancient decorative elements that are no longer visible in murals. On the other hand, the integrality of pigments is a major criterion for evaluating damage to the murals. In particular, irreparable damage is caused to the murals as the pigment layer peels. Therefore, it is crucial to identify and label the pigment layer peeling disease in mural conservation. Currently, machine learning is widely used for the identification and labeling of pigment layer diseases. The performance of four neural network algorithms for the pigment layer disease identification of the Mogao Caves is compared in [63]. According to the result, deep belief networks (DBNs) exhibit the best prediction performance under the strip noise condition compared with PLSR, PCA + SVM, and PCA + ANN. Compared with the other three models, DBN can exhibit the minimum RMSE of 0.2482, as well as the highest R-square of 0.5409 that is at least 4% higher than that of the other models. In conventional evaluation criteria, the number of diseases, size of the disease area, and other factors are used to reflect the severity of diseases, which has intuitiveness and limitations. In contrast, the comprehensive multi-index criterion can significantly improve classification accuracy over the method using single spectral information [13]. An extraction method for mural diseases based on encoder–decoder architecture has been proposed in [64]. It set up a modified U-Net network, which can preserve a part of the low-dimensional features through pyramid pooling. By comparing the modified U-Net network with the original one, it can be seen that the extraction accuracy is improved effectively, verifying the feasibility and superiority of the low-dimensional feature fusion network in identifying the pigment layer peeling disease. Moreover, hyperspectral image processing is also inspired by the application of neural networks in digital image processing. Compared with the digital images, the hyperspectral images contain spectral information. Hence, it deserves further exploration to combine the spectral information and the spatial information. Previous studies utilized the dual-channel dilated convolutional neural network [65], the multiple-kernel-based classifiers [66], the weighted joint collaborative representation [67,68], and the three-dimensional dilated convolution residual neural network [69] methods for hyperspectral image classification. It is found that hyperspectral image classification algorithms based on the fusion method may become popular in the future [70], while how to effectively combine the multidimensional information to improve the classification efficiency and accuracy will be a significant problem that needs to be solved.

4.3. Painting and Calligraphy

The main diseases of calligraphy and paintings include unclear text, blurred seals, unclear lines, and blurred faces. Hyperspectral imaging is suitable for the study of calligraphy and paintings, since it can identify information that is difficult to distinguish with the eyes, and can be also used for image enhancement and implicit information extraction.

There are quite a few investigations focusing on the hyperspectral recognition of paintings and calligraphy. In a case of Chinese painting and calligraphy “Sanqiutu” [71], the PCA method is used for information extraction, and 11 areas containing implicit information in the painting are captured with the analysis of spectral curves. Furthermore, a Qing Dynasty painting by Zhang Shibao is analyzed using the MNF method [72]. According to the analysis results, the image of the painting in MNF-Band 7 has a prominent response on the seal, while its image in MNF-Band 2 can highlight the ink line information of human figures. As shown in Figure 10, the image of the painting in MNF-Band 2 clearly presents the hidden information on the clothes after image enhancement processing. Meanwhile, the MNF method has also been used in the study of Tribute Envoys of the Song Dynasty [73]. Experimental results show that the MNF images can highlight the red parts like clothes and seals in the painting, and emphasize the cracks and repairs. In addition, MNF is used for the image enhancement of blurred seals in family genealogy, which can enhance the legibility and reading accuracy of the genealogical seals [74].

Hyperspectral imaging can uncover hidden information that has not been captured by previous equipment. As depicted in [8], databases from pulse-compression thermography (PuCT) and hyperspectral imaging can be merged together to inspect hidden information in paintings. PuCT is suitable for detecting splittings, cracks, and voids in the multi-layer structure by analyzing the painting’s thermal response via a thermal camera. Hyperspectral imaging is able to provide information concerning both pigments and pentimenti. It can be seen from the post-processing that the defected/hidden object detection can be further improved by using PCA and independent component analysis (ICA) on the original hyperspectral images and PuCT images [8]. The pentimenti are visible in both the first and third PCA, as well as in the first ICA, while the second PCA is able to show the brush strokes of the artist around 1480 nm. Therefore, by integrating the results obtained from PuCT and hyperspectral imaging, it is possible to build a satisfactory database on the hidden information of paintings, with its structure diagram shown in Figure 11.

In 2005, the POLA Museum of Art conducted a study on Picasso’s painting “Mother and Child by the Sea”. With X-ray imaging, the hidden female figure at the bottom of the painting can eventually be discovered, presumably a result of painting directly onto an older painting. In 2020, “Mother and Child by the Sea” is hyperspectrally analyzed [75]. According to its hyperspectral image, there are French newspaper texts printed onto the painting. These residual texts mainly resulted from using newspaper as the outer protection during the transportation of paintings, as shown in Figure 12. Then, the date of the newspaper is determined to be 18 January 1902, proving that “Mother and Child by the Sea” was completed around January 1902, unlike the previous common understanding that Picasso completed the painting after returning to Barcelona in 1901. This study shows the ability of hyperspectral imaging to uncover hidden information, which can help us to refine doubts left in history, and is also important for studying historical social activities. Hence, applications of hyperspectral imaging in cultural relic conservation may also bring breakthroughs in the study of history.

5. Challenges and Prospective Development

Based on the aforementioned investigations, it can be easily obtained that using hyperspectral imaging for cultural relic conservation still faces some critical challenges. First of all, the data dimensionality of hyperspectral images is very large. Without proper data dimensionality reduction, the available information is extremely limited, and consequently influences the performance of imaging. In practical applications, although the data dimensionality reduction methods are applied, there is still much redundant spectral information. Such a huge volume of data will make data processing heavily inefficient. Hence, more efficient data dimensionality reduction methods are still highly demanded in future investigations. Most significantly, data dimensionality is supposed to be reduced in accordance with the sensitive bands of different cultural relics, thereby trying to eliminate the spectral reflectance information that is irrelevant to the spectral differentiation analysis. Therefore, in further studies, more attention should be paid to the locations of the sensitive bands of various cultural relics.

Subsequently, spectral libraries are currently an important tool for analyzing the hyperspectral imaging data of cultural relics. However, due to the diversity of cultural relics, spectral libraries of cultural relics like pigments still need to be improved. The open access database from Europe, INFRA-ART Spectral Library, currently contains the spectral information of 819 pigment samples and keeps periodically updating. In the library, the Fourier transform infrared spectrum, Raman spectrum, short wavelength infrared reflectance spectrum, and XRF spectrum of pigments are available for free. Using these convenient spectral libraries can provide great assistance to research. On the other hand, not only spectral libraries of different kinds of cultural relics are essential, but also spectral data of the same kind from different periods are needed. By studying the spectral characteristics of cultural relics in different periods, the approximate time of the production of cultural relics can be rapidly obtained. Additionally, attention should be paid to spectral analysis without spectral libraries. In fact, it is really difficult to establish spectral libraries in some areas. Hence, performing spectral analysis without spectral libraries will remove the comparison process and greatly reduce the effort.

Moreover, some large outdoor cultural relics are severely affected by sunlight during the image capture process, making it very difficult to correct the image. The standard whiteboard method performs well in most cases subjected to uniform illumination. However, it cannot restore the shadow information of three-dimensional cultural relics subjected to intensive sunlight. Therefore, a feasible way is to print a 3D model of the cultural relic and take an image under the same lighting conditions as the whiteboard input for correction. A 3D model can well restore the reflectance of the cultural relic image under the effect of sunlight, but it should be noted that, since the source image and the whiteboard image are acquired from the cultural relic and the model, respectively, the correction algorithm, which is the calculation of image data from each pixel, requires a high degree of overlap between the location information of the two images. How to scale the two images to the same size is a significant problem, and more attention should be focused on how to use the hyperspectral camera to acquire equal-scale images of cultural relics and their 3D models, so as to reduce the error in data correction.

Ultimately, the conservation of cultural relics based on hyperspectral imaging requires deeper and more detailed investigations, and the combination of chemical analysis, Raman spectrum analysis, and other technical means can mutually support the experimental results and improve the accuracy of analysis.

6. Conclusions

This paper summarizes the imaging features, data processing methods, and applications in cultural relic conservation based on hyperspectral imaging, which has been found with high feasibility and unique advantages. Meanwhile, considering the difficulties of the conventional methods, the idea of improving the data dimensionality reduction and creating a standard spectral database for hyperspectral cultural relic conservation is proposed, so as to provide a reference with realistic significance. The non-contact and non-destructive hyperspectral imaging offers a safer solution for cultural relic conservation, and its property of “uniting image and spectrum” can enable to observe a great deal of hidden information.