Automatic Identification of Aurora Fold Structure in All-Sky Images

Abstract

Identification of small-scale auroral structures is key to searching for auroral events. However, it is impracticable for humans to manually select sufficient aurora events for statistical analysis, and it is also challenging for computers because of the non-rigid shape and fluid nature of auroras. Fold structure is the most common type of auroral small-scale structure, and its appearance is indicative of a variety of auroral events. This paper proposes a small-scale aurora structure identification framework to automatically detect aurora fold structures. First, the location and shape of auroras are identified based on a deep learning segmentation network. Then, the skeleton of the auroral shape is extracted to represent the trajectories of auroras. Finally, the proposed skeleton-based fold identification module (SFIM) can detect the aurora fold structure. To evaluate the effectiveness of the proposed method, we built an aurora fold structure sample dataset, namely F-dataset, containing 2000 images at 557.7 nm obtained by the all-sky imagers at Yellow River Station (YRS), Ny-Ålesund, Svalbard. Experimental results show that automatic identification can achieve good consistency with human perception. Statistical analysis of over 30,000 images shows that the fold occurrence has a distinct double-peak distribution at pre-noon and post-noon.

1. Introduction

Auroras are light emissions caused by the collision of ionized streams of charged particles with high-altitude atmospheric atoms and molecules. The aurora phenomenon is perceived as a convenient projection of effects from complex and energetic plasma processes of the outer magnetosphere [1]. The auroral behaviors are ascribed to various dynamic processes of the solar wind–magnetosphere interaction. Aurora observation is very important to studying space weather and solar wind–magnetosphere coupling [2]. Systematic observations of auroral patterns and their evolution provide a wealth of information on magnetospheric and solar–terrestrial electromagnetic activities. Two-dimensional optical emission distribution thus is shown to represent the particle pattern precipitated from the magnetosphere [3,4]. In recent years, optical observations have become important means of studying auroral phenomena. Compared with satellite observations observing the entire auroral oval, ground-based optical observations have a higher spatial resolution, which can reveal spatial and temporal variations in auroras [5]. Long-term continuous auroral observations provide an opportunity to study the aurora phenomena based on big data. Since the first search engine for the automatic retrieval of aurora images was developed by Syrjäsuo et al. in 2001 [6], researchers have conducted a growing number of studies on the automated analysis of aurora images and have achieved some outstanding results. These studies can be divided into two categories: auroral image classification and retrieval based on single images [1,7,8,9,10], and auroral image sequence classification and event detection based on image sequences [11,12,13,14,15,16].

Several of the analysis techniques mentioned above are based on the whole auroral image. However, when studying the occurrence regularity and co-occurrence probability of certain auroral events or local structures, it is inappropriate to consider all auroral activity in the all-sky field as a whole, on the one hand because multiple events may occur simultaneously, so independent auroral events cannot be counted correctly, and on the other hand, because the whole-image-based methods focus more on global features of auroral activity, the local auroral structures are often ignored. Even when auroral images are labeled as the same auroral type (e.g., arc), the spatial structures and motion patterns may be very different on small scales [17].

Many small-scale auroral structures can be observed by the all-sky imager (ASI), such as auroral vortex [18,19], curl [20], fragmented aurora-like emissions [21,22], etc. The energy distribution of precipitated electrons in an auroral small-scale structure provides the key to understanding the acceleration mechanism operating above the ionosphere [23]. These small-scale aurora structures move fast and occur in a short time [24,25]. Some artificial intelligence techniques are required to automatically identify and statistically study small-scale auroral structures. The fold structure is the most common type of auroral small-scale structure, and its appearance is indicative of a variety of auroral events. Unlike the narrow definition of mesoscale and clockwise folds (a type of aurora spiral), the fold structure here is considered to be a deformation pattern of an auroral arc, which is an indicator of multiple auroral forms. The size, number, and arrangement of the fold structures are very important parameters of auroral physics [26]. In this paper, we propose a method to automatically detect aurora fold structures in ASI images.

It is difficult to detect the fold structure using a supervised approach with a limited number of uncertain annotated samples. The supervised object detection techniques [27,28,29] require a large number of manual annotations for training. Unlike the objects to be detected in the previous studies, the scale and shape of local auroral structures in an ASI image are more random and varied. Additionally, the auroral emissions are inherently transparent and have no boundaries (the boundary of an aurora in an ASI image is determined by the preprocessing method). Therefore, it is almost impossible to obtain a large number of accurate manually annotated samples due to the uncertainty of the aurora fold structure (including variations in shape, scale, illumination, etc.) and the ambiguity of manual annotation. In addition, even if some relatively reliable manually labeled samples are obtained by spending a lot of time and effort on collecting them, the generalization ability of a supervised object detection technique is limited for auroral local structures. The objects in the previous study are mostly rigid or morphologically stable. The solar wind that causes the aurora is plasma with magnetic fluid properties, so the captured aurora image presents morphological instability. As for fold structure, there is no fixed template or pattern in an auroral image to be learned for object detection techniques.

In this paper, we propose an automatic identification method for aurora fold structure in ASI images without manual annotations. Based on the proposed method, aurora fold structures are retrieved from the ASI image dataset. We thereby analyzed the relationship between fold structure and dayside aurora appearance types and obtained the dayside occurrence distribution of aurora fold structures.

2. Methods

We propose a three-stage local auroral structure detection scheme based on skeleton extraction in ASI images. Instead of directly detecting local structures in ASI images, we first depict the morphological structure of the aurora, and then measure the fold structure based on the morphological structure. A three-stage approach is proposed to identify aurora fold structures, as shown in Figure 1.

2.1. Aurora Shape Knowledge Extraction

The aurora area is extracted from the background by image segmentation technology. Besides rich textures, low image contrast, and blurred boundaries, the intensity of auroral images varies over a wide range, and intensity inhomogeneity within a single ASI image is also very common. Due to these challenges of ASI auroral image, we resorted to the U-Net network, which has achieved great success in processing complex data [30,31,32]. We annotated hundreds of aurora images to train a U-Net segmentation network. The trained U-Net was used to achieve aurora image segmentation, and the network architecture with the parameters is shown in Figure 2.

Compared to the task of directly identifying fold structures, it is much easier to identify auroras in ASI images. There are much more aurora sample data than small-scale aurora fold structures, and large areas of aurora arcs are easier to annotate, which greatly mitigates the problem of sample imbalance. More importantly, subsequent identification results do not rely strongly on the accuracy of manual annotations of the aurora arc boundaries. Based on the manual annotations of auroras, the U-Net segmentation network was used at the first stage to find aurora areas in ASI images, which are also the approximate areas of auroral arc skeletons.

2.2. Skeleton Representation of Aurora

Similar to using a sketch of wave-like distortions to describe aurora vortex, fold, and curl in previous studies [33,34,35], the skeleton was used to represent auroral trajectory for identifying fold structure. It makes more sense to extract skeletal information about the auroral shape than to elaborately depict the boundaries of the aurora. Auroras observed in the sky at Earth’s high latitudes do not actually have distinct boundaries, and the sharp aurora boundaries in the processed ASI images depend on the dynamic range used to preprocess raw aurora image data. In contrast, the trajectory of the aurora is relatively stable, and its information can reflect a variety of important information about the aurora arc, such as the direction, distortion, number, length, and spacing of arcs. Figure 3a,b shows an example of extracting auroral skeletons on segmentation results, which contains the skeletonization process of the aurora, and the change in the pixel gray values from segmented aurora areas to skeletons within a fixed distance, respectively. Figure 3c shows some skeletonization results overlaid on ASI images.

Kerschnitzki et al. [36] extracted the medial axis of the three-dimensional foreground by iteratively removing voxels. We used this method to extract the skeleton on two-dimensional aurora images. Based on the obtained probability map of the aurora, we selected aurora pixels in ASI images. These aurora pixels were individually decided and iteratively removed until the one-pixel-thick skeleton was formed. Pixels were iteratively removed from the surface of the binary image if removing them did not change the connectivity of their 3 × 3 neighborhood and if they had more than one neighbor in their 3 × 3 neighborhood. The result is a one-pixel-thick medial axis skeleton that is homotopic to the aurora segmented from ASI images. The connectivity and topology of auroral arcs are preserved during skeletonization, and the resultant skeleton can portray the trajectory of auroral arcs.

2.3. Identification of Fold Structure

Based on the obtained skeleton, we identified aurora fold structures in a filtering way. Instead of detecting two-dimensional local structures in an ASI image, a two-dimensional filter works by moving through the skeleton to detect local fold structures. Specifically, a window of size $H$×$H$ was used to capture a local skeleton segment, which slides pixel by pixel over the entire skeleton. The identification process of the aurora fold structure is shown in Figure 4, i.e., the skeleton-based fold identification module (SFIM), which consists of four steps of skeleton evolution.

Given the original skeleton set $S=\left[s_1,s_2\dots \dots s_N\right]$ containing $N$ skeletons, to further eliminate noise and acquire a useful skeleton that is more likely to contain fold structures, $S$ is processed to the refined skeleton set $Z=\left[{\mathrm{z}}_1,z_2\dots \dots z_{N^{\prime }}\right]$ as $Z=T[S]$, and $N^{\prime }$ is less than $N$. The transformation $T[ ]$ includes truncating the skeleton with a zenith angle greater than $\theta$ and filtering the skeleton shorter than $L$. When the center of the filter traverses to the $k$th pixel of $z_i$, there are $M(k)$ pixel points $\left\{p_{i,k}^1,p_{i,k}^2\dots \dots p_{i,k}^{M(k)}\right\}$ in the $k$ th window. We propose RF to measure the pattern of the segmented skeleton, which is defined as

$$R_{i,k}=A_{i,k}-B_{i,k},$$

(1)

$$B_{i,k}={\Vert p_{i,k}^1,p_{i,k}^{M(k)}\Vert }_2,$$

(2)

where $A_{i,k}$ is skeleton length computed by counting the number of connected pixels in the skeleton, and $B_{i,k}$ is the Euclidean distance between two endpoints $p_{i,k}^1$ and $p_{i,k}^{M(k)}$ of the segmented skeleton within the $k$th window of $z_i$. $R_{i,k}$ is assigned to the $k$th pixel of $z_i$, which reflects the distortion degree of the segmented skeleton. With the sliding window, we can assign a value for each skeleton pixel, and finally obtain a quantified skeleton set $Q=\left[q_1,q_2\dots \dots q_{N^{\prime }}\right]$.

The fold structure is identified based on $Q$. The larger the value of $R_{i,k}$, the more likely the local area of the skeleton pixel is to form a fold structure. We can determine whether the segmented skeleton contains the fold structure via hard thresholding and derive a binarized skeleton set $F=[{\phi }_1,{\phi }_2\dots \dots {\phi }_{N^{\prime }}]$.

$${\phi }_i\left(n\right)=\{\begin{array}{l}1, q_i\left(n\right)\ge \tau , i=1,\dots ,N^{\prime }\\ 0, q_i\left(n\right)<\tau , i=1,\dots ,N^{\prime },\end{array}$$

(3)

where the $n$ th skeleton pixel of ${\phi }_i$ is assigned a logical value and $\tau$ is a preset threshold parameter. Due to the serious deformation at the edge of the lens during ASI image shooting, $\tau$ is set to a dynamic threshold that increases the further away from the image center. A segment of ${\phi }_i$ that is assigned consecutive positive values is considered to be a connected pixel set of a fold skeleton. Thus, the bounding box of the segmented skeletons with large distortions can be obtained. By adding a window width to the bounding box and mapping it back to the original ASI image, the position and size of the region containing the fold structures are obtained. The detailed procedure of the automatic identification algorithm for aurora fold structure is given in Algorithm 1.

Algorithm 1: Aurora fold structure automatic identification.

3. Data

3.1. Observation Data

The aurora image sequences used in this paper were acquired from the all-sky imagers at YRS, Ny-Ålesund, Svalbard, with the geographic coordinates of 78.92° N, 11.9°3 E, and corrected geomagnetic latitude of 76.24° and MLT≈UT + 3 h. Since 2003, the optical imaging system at YRS can make 24 h surveys of auroral emissions with a temporal resolution of 10 s in winter seasons. Thus, more than 1200 h of ASI images at three bands (R: 630.0 nm, G: 557.7 nm, and V: 427.8 nm) are obtained annually. Normally, the number of ASI images of YRS increases by more than one million frames per year. The size of the original ASI image is 512 × 512 pixels, and the average spatial resolution is 1.35 km as the zenith angle is from 0° to 90° (at 150 km). In this paper, ASI images at 557.7 nm acquired from December 2003 to February 2009 were used for the study of detecting fold structure. All images in this paper have been preprocessed and are 440 × 440 pixels.

3.2. F-Dataset

For quantitative evaluation of the proposed method, 2000 ASI images at 557.7 nm containing fold structures were selected by auroral experts from 2003 to 2009 observation data, which comprise the F-dataset. The F-dataset contains 2000 ASI images with various types of aurora fold structures, such as Figure 5a. By comparing the identification results by our method with those annotated by auroral experts, there are four possible identification outcomes: (1) true positive (TP)—the listed ASI image in the F-dataset is identified to have true fold structures; (2) false positive (FP)—the unlisted ASI image is identified as having fold structures; (3) false negative (FN)—the listed ASI image is identified as a non-fold; and (4) true negative (TN)—the unlisted event is identified as a non-fold.

In addition, we performed the proposed method on a large-scale aurora dataset. This way, we could statistically study the occurrence distribution and periodicity of the fold structures, and study the co-occurrence relationship between the small-scale structures and global appearance types, such as arc and corona. The auroral global appearance types are from a manually annotated ASI image dataset [1] (referred to as D-dataset), where 38,044 dayside (03:00–15:00 UT) ASI images captured in 19 days in the winter of 2003–2004 are annotated as arc (green), drapery corona (blue), radial corona (yellow), and hot-spot (red), as shown in Figure 5b. The detailed time and annotation of the D-dataset are shown in Figure 5c.

4. Experimental Results and Analysis

4.1. Setting and Preprocessing

In the segmentation of ASI images, we trained U-Net on NVIDIA GeForce RTX 3080 Ti graphics cards, and the framework for deep learning is Pytorch 1.9.1. In the transformation of original skeleton images, we adopted $\theta =$63$°$ and $L=$ 50. In the calculation of the RF for each pixel, we defined the size of the window as 71 × 71. In the identification of the fold structures, we set the thresholds $\tau =$ 20, 16, and 12 from the edge to the center of the image, respectively, to screen more suitable candidate fold structures.

To improve the performance of the proposed method in identifying fold structures from ASI images, we must preprocess the ASI images before implementing all algorithms, including dark current subtracting, gray-scale mapping, masking and cropping, and image rotation. These operations help to eliminate noise interference in original ASI images, confine the effective dynamic range, and calibrate the zenith angle and the magnetic north orientation.

We evaluate the aurora segmentation results in various challenges. Figure 6 shows four representative segmentation challenges, including (1) intensity inhomogeneity, (2) abundant texture structure, (3) small-scale aurora structure, and (4) low image contrast. From top to bottom in Figure 6, we show the segmentation results by (a) Otsu’s method, (b) an FCM-based image segmentation method, (c) a level set-based image segmentation method, and (d) U-Net, respectively. As shown in Figure 6, the U-Net outperforms the other three methods on four challenges. As shown, Otsu, FCM, and level set fail to handle the issue of low contrast. In the recognition of the continuity and complex structure of auroral arcs, the U-Net method has achieved good performance for various forms of auroras.

4.2. Performance of Aurora Fold Structure Identification

To demonstrate the effectiveness of the proposed automatic identification method for the aurora fold structure, we have conducted extensive experiments on the ASI image dataset. First, the performance of the RF-based skeleton pattern analysis was tested. Second, a quantitative evaluation was conducted in the F-dataset, with retrieval precision and recall computed to verify the accuracy of the proposed method. Finally, the retrieval of the fold structure in the large-scale dataset was analyzed in detail to illustrate the generalization ability and limitations of the method for various fold structures.

To evaluate the performance of RF-based skeleton pattern analysis, the RF-quantification skeleton is shown in Figure 7, where larger values on the skeleton indicate a greater distortion of the auroral arc. Based on the selected skeleton patterns, we can obtain fold structure regions, as shown in the red box on the ASI images. As shown, whether it is one or more fold structures (a), large-scale or small-scale (b), weak or strong auroral emission (c), and diffusion or discrete structures (d), our method can accurately identify the position and size of the fold structures.

A quantitative evaluation is conducted in the F-dataset to verify the accuracy of the proposed method. By comparing the automatic identification results with the expert annotations, we can calculate the retrieval precision and recall rate as

$$P_{precise}=\frac{TP}{TP+FP}$$

(4)

$$P_{recall}=\frac{TP}{TP+FN}$$

(5)

where $TP$ is the count of samples in the F-dataset which are identified as fold structures, $FN$ is the count of samples in the F-dataset which fail to be identified as fold structures, and $FP$ is the count of samples that are not in the F-dataset but are identified as fold structures. The precision rate indicates the probability of being correct in the detected fold structure samples, and the recall rate indicates the probability of finding fold structure samples in the F-dataset. The results are shown in

Table 1. The identification result of fold structures on F-dataset.

Annotation	Identification Result
	Positive	Negative
Positive	1271	729
Negative	137	863
Precise (%)	90.27
Recall (%)	63.55

, where the precise rate of identifying aurora folds is 90.27% and the recall rate is 63.55%. In total, 1271 ASI images in the F-dataset are correctly identified, which indicates that the proposed algorithm can identify the fold structures accurately. Meanwhile, 729 ASI images in the F-dataset are missed by our method, which indicates that our method is not suitable for some fold structures.

To illustrate the generalization ability and limitations of the method for various fold structures, we checked the retrieval results of the fold structure in the large-scale dataset. Figure 8a shows sixteen true-positive samples, which contain different types of auroras and are all detected with fold structures. Figure 8b shows eight ASI images without fold structures that are regarded as fold structures, which indicates that adjacent arcs are prone to false fold structures. Figure 8c shows eight missed fold structures. By overlaying the auroral skeleton on the ASI image, we found that the small-scale weak aurora structures are difficult to represent adequately with skeletons, which may be the reason why our method is unable to identify them.

4.3. Statistical Study of Aurora Fold Structures

To explore the distribution of fold structures in a large-scale aurora dataset and the co-occurrence relationship between fold structures and macroscopic auroral types, we applied the proposed method to the D-dataset to retrieve the aurora fold structure. A total of 5657 images were detected with fold structure, which are labeled as cyan in Figure 9.

As shown, the fold structures occur multiple times and continuously in a day, and may occur periodically and repeatedly in the auroral oval. In order to show the temporal distribution of the fold structures, the temporal axis was divided into 72 bins of 10 min durations. Images falling into each bin were counted. The occurrence distributions are then normalized by the total number of all images within the same time bin. The occurrence distribution of the aurora fold structure is shown in the bottom panel of Figure 10, which is compared with the distributions of the four discrete auroral types (arc, drapery corona, radial corona, and hot-spot, as shown in Figure 5). It is found that the fold occurrence has a distinct double-peak distribution. The first one is a small peak from 4:40 to 6:10 (UT), and the second one is a large peak from 12:10 to 14:10 (UT), which falls in the so-called pre-noon and post-noon occurrence peaks mainly contributed by the arc aurora.

5. Discussion and Conclusions

Aurora fold structures, the most common type of small-scale structures, are difficult to detect by whole-image-based techniques due to their variety of forms. In this study, we proposed an automatic identification method by combining deep learning and morphological knowledge to quickly identify, locate, and label aurora fold structures. Since the trained U-Net has a good segmentation performance on ASI images, it can help us to extract the shape of auroras from ASI images. Followed by iteratively skeletonizing auroral shape, the skeletons are extracted to represent the trajectory of auroras. For the identification step, the SFIM can identify aurora fold structures only by measuring the morphological feature of the skeleton without laborious labeling procedures on a large-scale dataset.

The experimental results showed that our method can identify the fold structure under different challenges, including different scales, quantities, brightness, and patterns. Compared with human annotations, our identification method achieves a high accuracy rate. On the large-scale dataset, we found the temporal occurrence peak of the aurora fold structure and the arc aurora is closer to the emergence pattern of the fold structure.

The ultimate purpose of identifying the small-scale aurora structure is to offer morphological interpretations of auroral types for aurora physics studies. In the future, we will conduct more aurora physics studies, such as the study of the statistics and classification of aurora spirals, based on the proposed local structure identification method. Meanwhile, the lower recall rate implies that some fold structures are undetected using our method, which may be caused by inaccurate skeletons. Our future work will also focus on improving the representation ability of skeletons for small-scale auroras with large curvature.