Underwater Hyperspectral Imaging System with Liquid Lenses

Abstract

The underwater hyperspectral imager enables the detection and identification of targets on the seafloor by collecting high-resolution spectral images. The distance between the hyperspectral imager and the targets cannot be consistent in real operation by factors such as motion and fluctuating terrain, resulting in unfocused images and negative effects on the identification. In this paper, we developed a novel integrated underwater hyperspectral imaging system for deep sea surveys and proposed an autofocus strategy based on liquid lens focusing transfer. The calibration tests provided a clear focus result for hyperspectral transects and a global spectral resolution of less than 7 nm in spectral range from 400 to 800 nm. The prototype was used to obtain spectrum and image information of manganese nodules and four other rocks in a laboratory environment. The classification of the five kinds of minerals was successfully realized by using a support vector machine. We tested the UHI prototype in the deep sea and observed a Psychropotidae specimen on the sediment from the in situ hyperspectral images. The results show that the prototype developed here can accurately and stably obtain hyperspectral data and has potential applications for in situ deep-sea exploration.

1. Introduction

As a combination of imaging and spectrometry, hyperspectral imaging can record the spectrum with high resolution for each spatial pixel of the image and construct a spectral image cube. The number of spectral bands can be in the range of several hundred, with a typical spectral spacing of less than 10 nm between each band. Hyperspectral imaging has been proven to be an effective marine remote sensing tool, enabling large-scale and continuous remote observation applications, such as ocean color inverse [1,2], marine ecological investigation [3,4], and disaster monitoring [5,6]. Recently, concerns for the marine ecosystem and environment conservation have heightened interest in underwater remote sensing. However, hyperspectral imaging systems based on airborne or satellite platforms can only cover a depth of several meters to tens of meters due to the dynamic conditions of the water-air interface, light, and atmospheric [7,8]. As a consequence, it is suitable only on the surface and in relatively shallow water.

The emergence of underwater hyperspectral imaging (UHI) technology provides a new perspective for underwater remote sensing. One significant benefit of the UHI system is that it eliminates the influence of the hybrid effect of water-air interfaces so that it can observe targets closer and more intuitively. Over the last decade, researchers have conducted extensive investigations pertaining to underwater hyperspectral imaging technology and developed several interesting prototypes [9,10,11,12]. Marine detection activities could be strongly supported by these devices, some scientifically significant, such as biological survey and habitat mapping [13,14,15], some environmentally important, such as the detection of microplastics [16], some more economically important, such as mineral detection [17,18], and some humanly important, such as the monitoring of pipelines and underwater archeology [19]. Underwater hyperspectral technology is becoming one of the most promising techniques in underwater optical exploration.

Push-broom hyperspectral imaging can achieve high-spatial and high-spectral resolution detection, with the advantages of no moving parts, compact structure, and easy integration. However, there are still restrictions to be solved in underwater applications. As a line scanning imaging method, it obtains a complete image cube relying on continuous exposure of every cross-section. The slight jitter in the scanning process may result in the displacement and deformation of the hyperspectral images. Meanwhile, due to platform movements and terrain fluctuations, the distance between the hyperspectral imager and the target of interest may be changed continually, which results in image blurring and reducing the accuracy of target identification.

There are several approaches on solving the first restriction. A relevant example is the use of navigation equipment equipped on underwater instrument carriers. The attitude and geometric could be corrected through the registration algorithm using geographic data recorded synchronously. In this paper, we are mainly centered on the second issue, i.e., clear imaging for the underwater hyperspectral imager. Common solutions usually rely on auxiliary sensors to measure the distance and keep the imager running at a fixed height from the target. For instance, Chennu et al. [20] developed a hyperspectral imaging and topographic surveying system for automated mapping of benthic habitats, whose altitude was controlled by the diver based on the measurement of an acoustic altimeter. Summers et al. [21] proposed a mini-ROV-UHI system for arctic macroalgal habitats mapping, which had a constant depth controlled at 0.5 m depth in the operating, with an altitude to the seafloor varying between 0.6–1.5 m. Unfortunately, there is little relevant work on the dynamic focusing method for the push-broom UHI imager, mainly for the reason that focusing has a large delay compared to the speed of distance variation. Thus, a UHI system with rapid and real-time focusing technology is required.

In this work, we developed a novel push-broom underwater hyperspectral imaging system prototype, which was an instrument deployed on underwater vehicle for the purpose of subsea detection. A liquid lens was employed as the telescope lens of the hyperspectral imager, which has the advantages of fast focus and small volume. The calibration results showed that the working spectral range was 400 nm to 800 nm and the spectral resolution correspondingly changed from 2.5 nm to 6.75 nm. In addition, a novel autofocus strategy for the HSI imager was presented to overcome many of challenges by using a focusing transfer method combined with the liquid lens. Instead of operating on the push-broom hyperspectral imager, this method achieved autofocus by using an RGB camera equipped with a liquid lens and transmitting the focusing information to the hyperspectral imager. The transfer factor of the two cameras was calibrated carefully, and the performances of the system were verified by imaging on mineral samples in laboratory. We also tested the prototype in the deep-sea field. The experiment results showed that the system had good imaging quality and reliability as a detection tool in underwater applications.

2. UHI System Design and Development

2.1. UHI System Design

The UHI system is an integrated instrument to be deployed on ROV, AUV, and HOV for the purpose of large-scale underwater surveys. Therefore, the issue of appropriate independence is considered attentively, which can potentially provide portability to suit different vehicles with respect to configuration adjustments for intended observation scenarios. In our prototype, the UHI system adopts a modular design and consists of three parts, which are the imaging subsystem, the sensing modules, and the control unit. Figure 1 presents the prototype and the encapsulated structure of the UHI system.

The imaging subsystem is a combination of the push-broom hyperspectral imager and the RGB camera. For the hyperspectral imager, a V10 spectrograph (Specim Ltd., Oulu, Finland), which consists of the slit, collimating optics, and grating, is used as the spectral splitting unit. In this case, the width of the slit is set to 30 μm with the view of the trade-off between the luminous flux and spectral resolution. The RGB camera has also been introduced into the imaging module, which can record the RGB imagery of the seafloor and provide the operator with a real-time view for better manipulation of the vehicles to track and aim at targets. Two electronically controllable focus lenses (Corning Inc., Glendale, AZ, USA) based on the liquid lenses are respectively utilized as the telescopes for both the hyperspectral imager and the RGB camera. There is nothing different, but the lens for the RGB camera (RGB lens) has a small focal length, resulting in a larger field of view than the lens for hyperspectral imaging (HSI lens). Such configuration aims to facilitate the post-processing of hyperspectral data, like geometric correction and image registration.

Attitude and ranging data are critical for underwater hyperspectral imaging to recover images from distortion and estimate the water influence. During underwater operation, the attitude of the UHI system may inevitably change on account of many factors, such as marine conditions and vehicle motion. Hence, a miniaturized attitude and heading reference system (AHRS) MTi-630 (Xsens Inc., Enschede, The Netherlands) are attached to the plate to capture orientation, acceleration, and inertial data. Besides, there are various commercial off-the-shelf ranging sensors available now, such as sonars and optical 3D ranging sensors, but they tend to be expensive, and these externally mounted sensors present some trouble with synchronizing hyperspectral data with sensor data. For these considerations, we modified and deployed a low-cost miniaturized phase-type single-point laser ranger inside the system to measure the distance roughly between the vehicle and targets. The laser ranger uses a 520 nm diode laser and has a range of 40 m with 3 mm accuracy in the air. All sensor data will be used for the calculation to compensate and calibrate the deviation to achieve the precise splicing of hyperspectral transects.

It is necessary to record sensor data while collecting each frame of the hyperspectral image. Thus, the cooperation of sensors and imager needs to be controlled in an orderly manner. A low-power-consumption single-board computer (SBC) is drawn into the control unit for data storage and operation control. The SBC also has an Arduino Leonard coprocessor, which generates a trigger signal for synchronous acquisition. The multi-serial port extension circuit is developed, which serves as a bridge to realize the interconnection between sensors and SBC. In data streams, hyperspectral images and the data of each sensor are recorded frame by frame, while RGB images are recorded every 20 frames. The skipped RGB images are aimed at calculating the focusing voltage in real time. All data is stored in a solid-state disk with a capacity of 2 TB, and images are synchronously transmitted to the computer on vehicles for further decision-making by operators. Following these strategies, the final control algorithm and software are developed.

Finally, the UHI system gathered with three parts is encapsulated in a pressure-tight shell. The shell is made of titanium alloy, and its rated depth for operation can reach up to 6000 m. With dimensions of 152 mm in outer diameter and 426 mm in length, the UHI system can be easily deployed on underwater vehicles. Two optional waterproof ports at the system’s tail allow 9–36 V direct-current input and ethernet communication with external vehicles. The detailed parameters of the UHI system are described in

Table 1. The detailed specifications and parameters of our UHI system.

Component	Category	Parameter
Imager	Imager	HSI	RGB
	F#	4	2.8
	Focal length	25 mm	16 mm
	FOV (transverse)	24.8°	38°
	Frame rate	100 Hz	75 Hz
	Resolution	1920 × 1	2448 × 2048
Sensor	Accuracy (roll/pitch/yaw)	0.2°/0.2°/1°
	Attitude data rate	1–400 Hz
	Ranging distance	0.2–5 m
Interface	Communication	100/1000 Mbps Ethernet
	Input voltage	9–36 VDC
	Power consumption	Max. 28 W (18 W typical)
Mechanism	Housing material	Titanium
	Size (Diameter × length)	152 × 426 mm
	Weight (air/water)	16/10 Kg
	MAX working depth	6000 m

.

2.2. Autofocus Strategy for UHI

Focusing distance will affect the clarity of the images. In some scenes that cannot be manually operated, the autofocusing process is usually completed with the help of sensors and algorithms. The autofocus methods are divided into three main types: searching on the look-up table (LUT) based on distance-aware [22], peak searching based on sharpness measurement [23], and combined method based on distance-aware and sharpness measurement [24].

Since one exposure of the push-broom hyperspectral imager only obtains the spatial information of one cross-section, in some scenes with insignificant spatial features, such as uniformly distributed sediments or targets with smooth surfaces, the method of peak searching sometimes oscillates and cannot converge quickly, resulting in misleading focus. Moreover, the method based on distance-aware may be ineffective because of the low surface reflectivity or complex terrain, leading to a wrong distance feedback. Thus, it is a challenging work for underwater hyperspectral imager to focus directly by using conventional methods in the underwater environment.

Here, we propose an autofocus strategy for the underwater hyperspectral imager by using the idea of focusing transfer based on liquid lens configuration. Regarding the hardware, the strategy requires cooperation between the RGB lens, the HSI lens, the RGB camera, and the hyperspectral imager. The liquid lenses are the core component, allowing switching the focusing distance rapidly by changing the voltage applied to the liquid lens [25]. The RGB camera serves as an assistant for the autofocus of the hyperspectral imager. Figure 2 shows the flowchart of the focusing transfer strategy for the UHI system. Firstly, measure the distance between the UHI system and the target by using the laser ranger, so the approximate interval of the focusing voltage (Vmin, Vmax) is determined. Then, the RGB camera executes the autofocusing procedure through the peak searching method based on sharpness measurement to obtain the focusing distance corresponding to the clearest image, that is, the focusing voltage of the RGB lens. The HSI lens finally focused by applying the RGB focusing voltage combined with the transfer factor to its voltage controller. This method only requires a simple calibration to determine the transfer factor between two liquid lenses. It should be noted that in some underwater areas with poor conditions, image preprocessing operations such as image enhancement, image noise reduction, and image defogging are required [26,27,28]. In addition, some other preprocessing methods, like visual saliency detection [29], can be used to assist in determining the area of interest for the autofocus procedure.

3. System Testing and Calibration

After the installation and adjustment of optical and mechanical structures, the UHI system was encapsulated in a waterproof pressure-tight shell. Then, related testing and calibration work was carried out to determine the performance of the UHI system. The key parameters, such as the central wavelength, spectral resolution, and other parameters related to relative and absolute spectral response, were determined through spectral and radiometric calibration. In addition, the transfer factor was also calibrated by the designed experiment setup.

3.1. Spectral Calibration

Spectral resolution refers to the minimum wavelength interval the imager can resolve, expressed by the full width at half maxima (FWHM) of the response spectrum. In the hyperspectral imager, the spectral resolution is affected by slit width, grating performance, pixel size, and other influencing factors introduced by alignment error. The experimental setup of spectral calibration is shown in Figure 3. The narrowband light generated by the halogen lamp and monochromator was expanded and collimated through the collimator, then entered into the UHI system and reached the corresponding position on the detector. The full spectrum calibration was completed by switching the wavelength of the narrowband light and recording images of the detector.

After calculation, with the 2× binning in spectral direction, the UHI system contained 480 bands covering the spectral range of 400–800 nm. Figure 4a shows the normalized response spectrum of the UHI system. There is a depression at 572 nm, which is caused by the long-pass filter in the spectrograph. Figure 4b presents the relationship among pixel location, spectral resolution, and wavelength. The pixel position and wavelength show a good linear relationship, and the spectral resolution is from 2.5 nm to 6.75 nm, with a mean value of 3.7 nm. From the perspective of the results above, the designed UHI system has a relatively high spectral resolution. In a sense, the UHI system can identify those materials with indicative spectral signatures from the mixtures.

3.2. Radiometric Calibration

The digital number (DN) is the pixel response of the input signal, which is associated with the exposure time and the spectral radiance of targets. Actually, due to some influence factors such as manufacturing defects, pixel location, and dark signal, even if the incident light is uniform, the DN of each pixel is different. Therefore, radiometric calibration is required to eliminate the influence and establish the relationship between DN and absolute spectral radiance.

Figure 5 shows the calibration instruments in the laboratory, including the integrating sphere, halogen lamp, and slit driven by the step motor. In radiometric calibration, the bad-pixel detection and dark signal calibration of the detector were carried out first. Then, the DN of the UHI system was measured at various exposure times from 1 ms to 250 ms under the constant illumination of the integrating sphere. Figure 6a shows the relationship between the DN of seven listed spectral bands and exposure times. Each line has good linearity, and the slope of lines reflects the responsiveness to the light intensity with different wavelengths.

In addition, another darkroom test was performed to test the response of the UHI system in different light intensity. The light intensity was adjusted by the slit inside, while a radiometer placed on the exit port of the integrating sphere was used to detect the spectral radiance. The DN was measured 50 times in one intensity at a fixed exposure time of 50 ms. Thus, the mean value per millisecond was calculated, and the corresponding relationship between the DN response and spectral radiance was finally established, which is shown in Figure 6b. Based on the results in Figure 6a,b, the spectral radiation characteristics of unknown targets can be obtained according to the DN response of the UHI systems, which is effective for eliminating the water influence and reconstructing the spectral reflectance in underwater imaging.

3.3. Autofocus Calibration

In the autofocusing process of dual-mode imaging, the RGB lens is adjusted to the optimal focusing voltage according to the sharpness measurement results of RGB images. Here, the RGB focusing voltage and transfer factor are combined and then transferred to the HSI lens. The relationship between the focusing distance and the corresponding focusing voltage of each lens requires measuring to determine the transfer factor.

Figure 7 shows the setup of autofocus calibration instruments. The UHI system was fixed on the lifter and placed at one end of the long-distance rail. A height-adjustable target with a focusing pattern was arranged at a variable location away from the UHI system. In the next step, changing the distance of the target to the UHI system continuously at the range of 20 cm to 250 cm and simultaneously searching for the optimal focusing voltage in the whole voltage range at an interval of 0.05 V. Finally, the relationship between focusing voltage and focusing distance was obtained to establish the look-up table of RGB and HSI lens.

Figure 8a shows the corresponding relationship between the focusing voltage and the focusing distance, which varies at the interval of 10 cm. As the distance increases, the optimal focusing voltage decreases rapidly and then tends to gently. Figure 8b shows the relationship between the focusing voltage of the RGB lens and the HSI lens. The curve-fitting result shows a good linear relationship, which follows the form of y = 1.033x − 0.2826 at the fitting coefficient of 0.99959. Among them, the slope of the fitted line is related to the attributes of both RGB and HSI lenses, while the intercept is affected by the installation position, for example, the front-to-back distance between the two lenses.

4. Experiment Result and Analysis

4.1. Autofocus Experiment for UHI Imager

A laboratory experiment was conducted to test the performance of the focusing transfer method. The experimental setup is shown in Figure 9. Two test targets were placed in the long-distance tank in tandem. First, the focusing voltage was adjusted to make target 1 in the clearest position. Then, the focusing transfer method was used to focus on target 2. In the experiment, the formula of Tenengrad (Tenenbaum gradient) was used to calculate the sharpness of the images, and the optimal focusing voltage was found by the method of golden section search.

The experimental results, which demonstrate a typical process of the focusing transfer method, are shown in Figure 10. Initially, the RGB and HSI lenses both focused on target 1. In the next step, the RGB lens started to execute the autofocusing procedure to change the focusing object from target 1 to target 2. Frames 1–6 present the intuitive changes in the sharpness of two targets. The RGB images of target 2 gradually became clear, while the hyperspectral images were always focused on target 1. However, once the focusing process was completed, the hyperspectral image was quickly focused on target 2 in the next frame. The experiment illustrates that good focusing results can be obtained through the focusing transfer from the RGB lens to the HSI lens.

The two lenses are mounted as close as possible to ensure an appropriate field of view overlap in the observation area of the two lenses. The distance between the lenses is preferably negligible compared to the distance between the lenses and the observation targets to avoid the influence of the reduction of image quality on focusing at a large viewing angle. The benefit of focus transfer is that the hyperspectral imager will not capture invalid images used for sharpness measurement during continuous acquisition. Besides, the time to complete one autofocusing process has greatly shortened based on the liquid lens’s rapid response and adjustment capability. It should be pointed out that the combined method used in the focusing transfer method is replaceable. Any method that can effectively achieve autofocusing should be considered. Continuous autofocusing can be realized during the acquisition by appropriately setting the decision conditions and thresholds.

4.2. Hyperspectral Imaging Experiment

The manganese nodule samples used in this work were obtained from the seamount areas in the western Pacific Ocean at depths of 4000 to 5000 m. They are approximately 3–5 cm in shape size and between black and brown in surface color. Besides, four common rocks, such as granite, gabbro, andesite, and basalt, were chosen to enrich the abundance of sample types. The sediments were a mixture of mud and sand collected nearby the coastal areas.

The mineral samples were placed at the bottom of the black tank, and hyperspectral images were taken at a distance of 30 cm from the bottom using the designed UHI system under the illumination of two 50 W halogen lamps. Figure 11 shows the pseudo-RGB image of the mineral samples.

The properties of material, such as shape, transparency, and surface roughness, affect the reflection of its surface. The reflectance spectrum of materials immersed in water tends to be different from that in the air due to absorption and interface effects. Figure 12 shows the corrected reflectance spectrum of mineral samples from 420 nm to 800 nm. Spectral bands before 420 nm were eliminated due to the low signal-to-noise ratio. Although there is a lack of indicative absorption signatures, the manganese nodules still reflect trends distinct from other samples in the reflectance spectrum. Figure 13 shows the classification results of the mineral samples based on the support vector machine (SVM) method. Since the high spatial resolution characteristics brought by the close observation, there are some misclassifications on the surface of the mineral samples. Gabbro and basalt have similar chemical compositions, and their reflectance spectrum is close to andesite, which leads to the low classification accuracy among the three samples. The signature of manganese nodules differs from all other samples, allowing them to be classified almost accurately.

4.3. Deep-Sea Field Test for the UHI Prototype

We carried out a pilot underwater experiment to further test the practical performance of the UHI system, in June 2022. The test site was located at the seamount with a depth of more than 4000 m in the northwestern Indian Ocean. The submersible used in the test was the Jiaolong human-occupied vehicle (HOV), which has a maximum operating depth of 7000 m. Figure 14 shows the setup and communication test of the UHI system with the submersible. The UHI system was installed on the front of the HOV to obtain power and exchange data from the submersible through the power and ethernet connector.

During the underwater experiment, the push-broom scanning was performed along the vertical direction of the slit under the irradiation of the high-power light source mounted on the HOV. The orientation during scanning was controlled appropriately to decrease the distortion. The submersible traveled at about 0.5 m/s, while the UHI system recorded at 50 frames per second. Figure 15 shows a pseudo-RGB image of seafloor sediment from a downward viewing perspective. There are some undulating dunes and tiny holes in the seafloor, while a Psychropotidae specimen lies on the right.

5. Conclusions

We designed an integrated push-broom underwater hyperspectral imaging system for deep-sea applications. The working spectral range was 400 nm to 800 nm with the spectral resolution changing from 2.5 nm to 6.75 nm. We proposed a novel autofocus strategy for the hyperspectral imager based on the focusing transfer method. With the assistance of RGB camera and the liquid lens, it is efficient and rapid for the hyperspectral imager to find the optimal focus distance. We detected the spectral images of deep-sea manganese and other four kinds of rocks in the laboratory environment. The result showed that the hyperspectral imager had good imaging quality and could identify targets accurately. We tested the prototype in the deep-sea field with the depth of over 4000 m and achieved images of seafloor sediment successfully. Future research on the system will focus on improving the automation level and reducing the cost of human intervention, so as to apply the system to deep-sea explorations.