Inverse Airborne Optical Sectioning

Abstract

We present Inverse Airborne Optical Sectioning (IAOS), an optical analogy to Inverse Synthetic Aperture Radar (ISAR). Moving targets, such as walking people, that are heavily occluded by vegetation can be made visible and tracked with a stationary optical sensor (e.g., a hovering camera drone above forest). We introduce the principles of IAOS (i.e., inverse synthetic aperture imaging), explain how the signal of occluders can be further suppressed by filtering the Radon transform of the image integral, and present how targets’ motion parameters can be estimated manually and automatically. Finally, we show that while tracking occluded targets in conventional aerial images is infeasible, it becomes efficiently possible in integral images that result from IAOS.

1. Introduction

Higher resolution, wide depth of field, fast framerates, high contrast, or signal-to-noise ratio can often not be achieved with compact imaging systems that apply narrower aperture sensors. Synthetic aperture (SA) sensing is a widely recognized technique to achieve these objectives by acquiring individual signals of multiple or a single moving small-aperture sensor and by computationally combining them to approximate the signal of a physically infeasible, hypothetical wide aperture sensor [1]. This principle has been used in a wide range of applications, such as radar [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28], telescopes [29,30], microscopes [31], sonar [32,33,34,35], ultrasound [36,37], lasers [38,39], and optical imaging [40,41,42,43,44,45,46,47].

In radar, electromagnetic waves are emitted and their backscattered echoes are recorded by an antenna. Electromagnetic waves at typical radar wavelengths (as compared with the visible spectrum) can penetrate scattering media (i.e., clouds, vegetation, and partly soil) and are quite useful for obtaining information in all weather conditions. However, acquiring high spatial resolution images would require an impractically large antenna [2]. Therefore, since its invention in the 1950s [3,4], Synthetic Aperture Radar (SAR) sensors have been placed on space-borne systems, such as satellites [5,6,7,8], planes [9,10,11], and drones [12,13] in different modes of operation, such as strip-map [11,14], spotlight [11,14], and circular [10,14] to observe various sorts of phenomena on Earth’s surface. These include crop growth [8], mine detection [12], natural disasters [6], and climate change effects, such as the deforestation [14] or melting of glaciers [7]. Phase differences of multiple SAR recordings (interferometry) have even been used to reconstruct depth information and enables finer resolutions [15].

Analogous to SAR (which utilizes moving radars for synthetic aperture sensing of widely static targets), a technique known as Inverse Synthetic Aperture Radar (ISAR) [16,17,18] considers the relative motion of moving targets and static radars for SAR sensing. In contrast to SAR (where the radar motion is usually known), ISAR is challenged by the estimation of an unknown target motion. It often requires sophisticated signal processing and is often limited to sensing one target at a time, while SAR can image large areas and monitor multiple (static) targets simultaneously [17,18]. ISAR has been used for non-cooperative target recognition (non-stationary targets) in maritime [19,20], airspace [21,22], near-space [23,24], and overland surveillance applications [25,26,27,28]. Recently, spatially distributed systems and advanced signal processing, such as compressed sensing and machine learning, have been utilized to obtain 3D images of targets, target’s reflectivity, and more degrees of freedom for target motion estimation [27,28].

With Airborne Optical Sectioning (AOS) [48,49,50,51,52,53,54,55,56,57,58,59,60], we introduced an optical synthetic aperture imaging technique that captures an unstructured light field with an aircraft, such as a drone. We utilized manually, automatically [48,49,50,51,52,53,54,55,56,58,59], or fully autonomously [57] operated camera drones that sample multispectral (RGB and thermal) images within a certain (synthetic aperture) area above occluding vegetation (such as forest) and combined their signals computationally to remove occlusion. The outcome is a widely occlusion-free integral image of the ground, revealing details of registered targets while unregistered occluders above the ground, such as trunks, branches or leaves disappear in strong defocus. In contrast to SAR, AOS benefits from high spatial resolution, real-time processing rates, and wavelength-independences, making it useful in many domains. So far, AOS has been applied to the visible [48,59] and the far-infrared (thermal) spectrum [51] for various applications, such as archeology [48,49], wildlife observation [52], and search and rescue [55,56]. By employing a randomly distributed statistical model [50,57,60] the limits of AOS and its efficacy with respect to its optimal sampling parameters can be explained. Common image processing tasks, such as classification with deep neural networks [55,56] or color anomaly detection [59] are proven to perform significantly better when applied to AOS integral images compared with conventional aerial images. We also demonstrated the real-time capability of AOS by deploying it on a fully autonomous and classification-driven adaptive search and rescue drone [56]. Yet, the sequential sampling nature of AOS when being used with conventional single-camera drones has limited its applications to recover static targets only. Moving targets lead to motion blur in the AOS integral images, which are nearly impossible to classify or to track.

In [59], we presented a first solution to tracking moving people through densely occluding foliage with parallel synthetic aperture sampling supported by a drone-operated, 10 m wide, 1D camera array (assembling 10 synchronized cameras). Although feasible, such a specialized imaging system is in most cases is impractical as it is bulky and difficult to control.

Being inspired by the principles of ISAR for radar, in this article we present Inverse Airborne Optical Sectioning (IAOS) for detecting and tracking moving targets through occluding foliage (cf. Figure 1b) with a conventional, single-camera drone (cf. Figure 1c). As with ISAR, IAOS relies on the motion of targets being sensed by a static airborne optical sensor (e.g., a drone hovering above forest) over time (cf. Figure 1a) to computationally reconstruct an occlusion-free integral image (cf. Figure 1d). Essential for an efficient reconstruction is the correct estimation of the target’s motion.

In this article, we make four main contributions: (1) We introduce the principles of IAOS (i.e., inverse synthetic aperture imaging) in Section 1 and Section 2. (2) We explain how the signal of occluders can be further suppressed by filtering the Radon transform of the image integral in Section 2.1 (cf. Figure 1e). (3) We present how a target’s motion parameters can be estimated manually and automatically in Section 2.1 and Section 2.2. (4) Finally, we show that while tracking occluded targets in conventional aerial images is infeasible, it is efficiently possible in integral images that result from IAOS in Section 3.

2. Materials and Methods

All field experiments were carried out in compliance with the legal European union Aviation Safety Agency (EASA) flight regulations, using a DJI Mavic 2 Enterprise Advanced, over dense broadleaf, conifer, and mixed forest, and under direct sunlight as well as under cloudy weather conditions. Free flight drone operations were performed using the DJI’s standalone smart remote controller with DJI’s Pilot application. RGB videos of resolution 1920 × 1080 (30 fps) and thermal videos of resolution 640 × 512 (30 fps) were recorded on the drone’s internal memory, and were processed offline after landing. For vertical (top-down, as in Figure 3) scans the drone was hovering at an altitude of about 35 m AGL. For horizontal scans (sideways, as in Figure 4) the drone was hovering at a distance of about 10 m away from the vegetation. For quicker processing, we extracted a selection of 1–5 fps from the acquired 30 fps thermal videos using FFmpeg python bindings. Offline processing included intrinsic camera calibration (pre-calibrated transformation matrix computed using MATLAB’s camera calibrator application) and image un-distortion/rectification using OpenCV’s pinhole camera model (as explained in [48,55]). The undistorted and rectified images were cropped to a field of view of 36° and a resolution of 1024 × 1024 px. Image integration was achieved by averaging the pre-processed images being registered based on manually or automatically estimated motion parameters, as explained in Section 2.1 and Section 2.2. Radon transform filtering [61,62,63] (also explained in Section 2.1 and Section 2.2) was implemented in Mathworks’ MATLAB R2022a.

2.1. Manual Motion Estimation

If the target’s motion parameters (i.e., direction, θ [°] and speed, s [m/s]) are known and assumed to be constant for all time intervals, the captured images can be registered by shifting them accordingly to θ and s. Thereby, θ can directly be mapped to the image plane, while s must be mapped [m/s] to [px/s] (which is easily possibly after camera calibration and knowing the drone’s altitude). By averaging the registered images results in an integral image that shows the target in focus (local motion of the target itself, such as arm movements of a walking person, lead to defocus) while the misregistered occluders vanish in defocus.

Large occluders that are shifted in direction θ while being integrated appear as linear directional blur artifacts in the integral image (cf. Figure 1d). Their signal can be suppressed by filtering (zeroing out) the Radon transform of the integral image I(θ,s) in direction θ (+/− an uncertainty range that considers local motion non-linearities of the occluders, such as movements of branches caused by wind, etc.). The inverse Radon transform (filtered back projection [63]) of the filtered sinogram results in a new integral image with suppressed signal of the directionally blurred occluders (cf. Figure 1e). This process is illustrated in Figure 2, and can be summarized mathematically with:

$$I^{\prime }\left(\mathsf{\theta },s\right)={\mathrm{Rf}}^{-1}\left(\mathrm{F}\left(\mathrm{Rf}\left(I\left(\mathsf{\theta },s\right)\right),\mathsf{\theta }\right)\right),$$

(1)

where F is the filter function which zeros out coefficients at angle θ (+/− uncertainty range) in the sinogram.

One way of estimating the correct motion parameters is by visual search (i.e., θ and s are interactively modified until the target appears best focused in the integral image). Exploring the two-dimensional parameter space within proper bounds is relatively efficient if the motion can be assumed to be constant. Sample results are presented in Section 3. See also Supplementary Video S1 for an example of manual visual search for the motion parameters of results shown in Figure 3k. In case of non-linear motion, the motion parameters must be continuously and automatically estimated. A manual exploration becomes infeasible in this case.

2.2. Automatic Motion Estimation

Automatic estimation of motion parameters requires an error metric which is capable of detecting improvement and degradation in visibility (i.e., focus and occlusion) for different parameters. Here, we utilize simple gray level variance (GLV) [64] as an objective function. We already proved in [53] that, in contrast to traditionally used gradient-, Laplacian-, or wavelet-based focus metrics [65], GLV does not rely on any image features and is thus invariant to occlusion. In [54] (see also Appendix A), we demonstrated that the variance of an integral image is:

$$Var\left[I\right]=\frac{D\left(1-D\right)({\left({\mu }_o-{\mu }_s\right)}^2)+D{\sigma }_o^2+\left(1-D\right){\sigma }_s^2}{N}+{\left(1-D\right)}^2(1-\frac{1}{N}){\sigma }_s^2,$$

(2)

where $\mathrm{D}$ is the probability of occlusion, while ${\mathsf{\mu }}_{\mathrm{o}}$,${\sigma }_o^2$ and ${\mathsf{\mu }}_s,{\sigma }_s^2$ are the statistical properties of occlusion and the target signal, respectively.

Integrating $N$ individual images with optimal motion parameters results in an occlusion-free view of the target’s signal whereas the signal strength of the occluders reduces and disappears in strong defocus. To further suppress occluders, we used Radon filtering [61,62,63] as described in Section 2.1. However, we now utilize the linearity property of the Radon transform which states that:

$$Rf\left({{\displaystyle \sum }}_i{\alpha }_i{I}_i\right)={{\displaystyle \sum }}_i {\alpha }_{i}Rf(I_i).$$

(3)

Thus, instead of filtering the integral image I(θ,s), as explained in Equation (1), we apply Radon transform filtering to each single image $I_i$ before integrating it.

For automatic motion parameter estimation, we registered the current integral image I (integrating $I_1^{\prime }\dots I_{i-1}^{\prime }$) to the latest (most recently recorded) inverse Radon transformed filtered image $I_i^{\prime }={\mathrm{Rf}}^{-1}\left(\mathrm{F}\left(\mathrm{Rf}(I_i\right),\mathsf{\theta }\right))$ by maximizing $\mathrm{Var}\left[I\right]$ while optimizing for best motion parameters ($\mathsf{\theta },s$). Deterministic-global search, DIRECT [66] (as implemented Nlopt [67]), was applied for optimization. Consequently, we considered each discrete motion component between two recorded images and within the corresponding imaging time (e.g., 1/30 s for 30 fps) to be piecewise linear. The integration of multiple images, however, can reveal and track a non-linear motion pattern where ($\mathsf{\theta },s$) vary in each recording step. Sample results are presented in Section 3 and in Supplementary Videos S2 and S3.

3. Results

Figure 3 presents results from field studies of IAOS with manual motion estimation, as explained in Section 2.1. Images are recorded top-down, with the drone hovering at a constant position above conifer (Figure 3a–l), broadleaf (Figure 3m–o), and mixed (Figure 3p–r) forest. Estimated motion parameters of hidden walking people were: 118°, 0.5 m/s (Figure 3a–l), 108°, 0.6 m/s (Figure 3m–o), and 90°, 0.6 m/s (Figure 3p–q).

Figure 4 illustrates an example with the drone hovering at a distance of 10 m in front of dense bushes (at an altitude of 2 m, recording horizontally). The hidden person is walking from right to left at 260° with 0.27 m/s (both manually estimated).

Figure 5 illustrates two examples for automatic motion estimation, as explained in Section 2.2, with the drone hovering at an altitude of 35 m and a hidden person walking through dense forest.

For tracking, moving targets are first detected by utilizing background subtraction based on Gaussian mixture models [68,69]. The resulting foreground mask is further processed using morphological operations to eliminate noise [70,71]. Subsequently blob analysis [72,73] detects connected pixels corresponding to each moving target. Association of detections in subsequent frames is entirely based on motion where the motion of each detected target is estimated by a Kalman filter. The filter predicts the target location in subsequent frame (based on previous motion and associated motion model) and then determines the likelihood of assigning the detection to the target.

For comparison, we applied the above tracking approach to both: the sequence of captured single thermal images and to the sequence of integral images computed from the single images, as described in Section 2.2. For each case, tracking parameters (such as minimum blob size, max. prediction length, number of training images for background subtraction) were individually optimized to achieve best possible results.

While tracking in single images leads to many false positive detections becoming practically infeasible, tracking in integral images results in clear track-paths of a single target. Estimated mean motion parameters were: 291°, 0.82 m/s (Figure 5a–c), 309°, 0.16 m/s for the first leg, and 241°, 0.41 m/s for the second leg (Figure 5d–f). See Supplementary Videos S2 and S3 for dynamic examples of these results.

4. Discussion and Conclusions

In this article we presented Inverse Airborne Optical Sectioning (IAOS), an optical analogy to Inverse Synthetic Aperture Radar (ISAR). Moving targets, such as walking people, that are heavily occluded by vegetation can be made visible and tracked with a stationary optical sensor (e.g., a hovering camera drone above forest). We introduced the principles of IAOS (i.e., inverse synthetic aperture imaging), explained how the signal of occluders can be further suppressed by filtering the Radon transform of the image integral, and presented how targets’ motion parameters can be estimated manually and automatically. Furthermore, we showed that while tracking occluded targets in conventional aerial images is infeasible, it is efficiently possible in integral images that result from IAOS.

IAOS has several limitations: We assume that local motion of occluders and of the drone (e.g., caused by wind) is smaller than the motion of the target. Small local motion of the target itself, such as individual moving body parts, appear blurred in integral images. Moreover, the field of view of a hovering drone is limited and moving targets might be out of view quickly. In the future, we will investigate how drone movement being adapted to target movement can increase field of view and reduce blur of local target motion. This corresponds to a combination of IAOS (i.e., occlusion removal by registering target motion) and classical AOS (i.e., occlusion removal by registering drone movement). Furthermore, results of Radon transform filtering have artifacts that are due to under-sampling; higher imaging rates can overcome this. The blob-based tracking approach applied for proof-of-concept is very simple; more sophisticated methods achieve superior tracking results. See supplementary Videos S2 and S3 for dynamic examples of these results. However, we believe that tracking in integral images will always outperform tracking in conventional images.