Superpixel-Based Segmentation of Polarimetric SAR Images through Two-Stage Merging

Abstract

Image segmentation plays a fundamental role in image understanding and region-based applications. This paper presents a superpixel-based segmentation method for Polarimetric SAR (PolSAR) data, in which a two-stage merging strategy is proposed. First, based on the initial superpixel partition, the Wishart-merging stage (WMS) simultaneously merges the regions in homogeneous areas. The edge penalty is combined with the Wishart energy loss to ensure that the superpixels to be merged are from the same land cover. The second stage follows the iterative merging procedure, and applies the doubly flexible KummerU distribution to better characterize the resultant regions from WMS, which are usually located in heterogeneous areas. Moreover, the edge penalty and the proposed homogeneity penalty are adopted in the KummerU-merging stage (KUMS) to further improve the segmentation accuracy. The two-stage merging strategy applies the general statistical model for the superpixels without ambiguity, and more advanced model for the regions with ambiguity. Therefore, the implementing efficiency can be improved based on the WMS, and the accuracy can be increased through the KUMS. Experimental results on two real PolSAR datasets show that the proposed method can effectively improve the computation efficiency and segmentation accuracy compared with the classical merging-based methods.

1. Introduction

A Polarimetric synthetic aperture radar (PolSAR) system transmits and receives electromagnetic waves with four polarization combinations, therefore it can provide more useful information and better characterization of the earth surface than single-polarization cases. However, the speckle noise inherent within PolSAR data hinders image interpretation and specific applications. Speckle filtering [1] and spatial information extraction [2] are usually applied to alleviate the effect of speckle noise. Recently, an object/region-based processing scheme is drawing increasing attention in various PolSAR applications including land use/land cover classification [3,4,5,6,7], change detection [8,9], target detection [10,11], wetland monitoring [12,13], and so on. As a preprocessing step, region generation/segmentation divides the image into distinct and self-similar pixel groups, and it is beneficial to reduce the effect of speckle noise and can accelerate the subsequent procedures to a great extent.

Various kinds of methods have been proposed for PolSAR image segmentation, in both pixel-based and region-based manners. Among them, one type of method applies Markov random field (MRF) model to impose a spatial regularity constraint on the segmentation. For example, the MRF framework was coupled with the cluster analysis in the tensor space [14], and the final segmentation result was obtained using a graph-cut-based method. A region-based segmentation method was proposed using the Wishart MRF (WMRF) model [4]. Region merging is another commonly used technique in PolSAR segmentation. Cao et al. [15] applied an agglomerative hierarchical clustering method to over-segment the image, and then iteratively merged the two regions with the minimum distance which is calculated based on the Wishart test statistic. In [16], a region-based unsupervised PolSAR segmentation and classification algorithm was proposed, which incorporates region growing and an MRF edge strength model. The statistical region merging (SRM) algorithm has been generalized for PolSAR image segmentation by taking the multiplicative property of speckle noise into account [17]. In addition, the binary partition tree (BPT) [18] can also be used for PolSAR segmentation. By iteratively merging the most similar neighboring regions, a BPT can be constructed accordingly. On this basis, Salembier et al. [19] investigated several optimum graph-cut techniques to prune the BPT of PolSAR images, resulting in the partitioned BPT which represents the final segmentation result. Besides MRF and region merging, other promising tools and approaches can also be employed for PolSAR segmentation. For example, a spatially adaptive segmentation framework was proposed using a wedgelet representation approach [20]. The spectral graph partitioning [21] technique first segments the image into regions with the aid of contour and spatial proximity information, and then groups them through spectral clustering based on the Wishart distance between neighboring segments.

Even though many segmentation methods have been proposed, as stated above, these methods mainly explore the statistical characteristics of PolSAR data based on the complex Wishart distribution [22] which is derived according to the Gaussian assumption of scattering vectors. The Wishart distribution can well characterize the backscatters in homogeneous areas. However, recent studies [23,24,25] reveal that non-Gaussian distributions can better model the scattering vector statistics, compared to the complex Gaussian distribution. The non-Gaussianity of PolSAR data is generally represented by the product model, which states that, under certain conditions, the scattering vector results from the product between the square root of a scalar random variable (texture) and an independent Gaussian random vector (speckle). Based on the product model, various non-Gaussian statistical models of PolSAR data have been proposed by adopting different textural distributions. The K distribution [26] and G⁰ distribution [27] are with one texture parameter, and they are derived based on Gamma distributed and inverse Gamma distributed texture, respectively. Bombrun et al. [24] proposed to generalize these two kinds of distributions with the two-parameter KummerU distribution, which is derived from the Fisher distributed texture. The KummerU distribution is more flexible since it includes the Wishart, K, and G⁰ distributions as asymptotic cases [28].

Based on the non-Gaussian statistical models, several studies have been conducted for PolSAR image segmentation [23,28,29]. An unsupervised contextual segmentation method was proposed in [23] by incorporating the K distribution with the MRF framework. Similarly, the method in [28] applies the doubly flexible KummerU distribution to characterize the PolSAR data, and achieves automatic clustering via the expectation maximization (EM) method. These two methods realize the segmentation and clustering of the pixels at the same time, whereas it is difficult to determine the number of classes. In addition, they are implemented at the pixel level, resulting in some misclassifications in the final result due to the inherent speckle noise. Beaulieu and Touzi [30] proposed a hierarchical segmentation algorithm for PolSAR data based on stepwise optimization, and this method has been adapted to the KummerU-distributed covariance matrix and vector data in [24] and [29], respectively.

In this paper, considering the merits of KummerU distribution and superpixel, we propose a novel segmentation framework for the multi-look PolSAR data. Superpixel generation of PolSAR images is recently studied in several studies [31,32,33], and the obtained superpixels have been incorporated into object-oriented classification methods, e.g., in [5,34]. This paper aims at segmenting the PolSAR data on the level of superpixels instead of pixels, thereby it can effectively reduce the effect of speckle noise. Owing to the advantage of the KummerU distribution in PolSAR data characterization, more reliable results would be obtained. However, the high time consumption with respect to the KummerU distribution should also be considered in the segmentation method. Based on the classical iterative merging procedure in [24], this paper proposes a two-stage merging strategy, which includes the Wishart-merging stage (WMS) and the KummerU-merging stage (KUMS). Using the Wishart criterion, the WMS efficiently merges the superpixels in homogeneous areas. Afterwards, the KUMS takes the advantage of the KummerU distribution, and executes the iterative merging procedure. Then the processing time can be greatly reduced when implementing on the resultant regions from WMS. In addition, the edge penalty and the proposed homogeneity penalty are also applied in the merging criterion to further improve the segmentation accuracy.

The remainder of this paper is organized as follows. First, the necessary theoretical background of PolSAR data is presented, especially the statistical models and the parameter estimation procedure with respect to KummerU distribution. Then, the proposed segmentation framework is provided in detail, based on the initial superpixel partition. The performance of the proposed method is demonstrated on two real PolSAR data sets, and the conclusion is given at the end.

2. Statistical Distributions of PolSAR Data

For a reciprocal medium illuminated by the monostatic SAR, the polarimetric scattering information can be described by the target scattering vector $k$ using linear basis;

$$k={\left[S_{\mathrm{HH}},\sqrt{2}{S}_{\mathrm{HV}},S_{\mathrm{VV}} \right]}^{\mathrm{T}},$$

(1)

where $S_{\mathrm{HV}}$ is the scattering element of horizontal transmitting and vertical receiving polarization, and the superscript $\mathrm{T}$ means the matrix transpose. Usually, the vector data are multi-looking processed for speckle reduction, and the obtained PolSAR data are represented by the polarimetric covariance matrix $C$ as:

$$C=\frac{1}{L}{\displaystyle \sum }_{n=1}^L{k}_n{k}_n^{†},$$

(2)

where $L$ is the nominal number of looks, and $†$ denotes the complex conjugate transpose. In this section, the statistical models of multi-look complex (MLC) covariance matrix data are introduced as the basic theory of the proposed segmentation method.

2.1. Wishart Distribution

In homogeneous areas, the scattering vector data are modeled by a multivariate zero-mean circular Gaussian distribution, and then the MLC covariance matrix follows the complex Wishart distribution [22]. The probability density function (PDF) of the covariance matrix $C$ is given by:

$$p(C|{{\displaystyle \sum }}^{},L)=\frac{L^{dL}{\left|C\right|}^{L-d}\exp (-L\cdot \mathrm{Tr}({{{\displaystyle \sum }}^{}}^{-1}C))}{K\left(L,d\right){|{{\displaystyle \sum }}^{}|}^L},$$

(3)

$$K\left(L,d\right)={\mathsf{\pi }}^{d\left(d-1\right)∕2}{\displaystyle \prod }_{i=1}^d\mathsf{\Gamma }\left(L-i+1\right),$$

(4)

where $d$ denotes the dimension of $k$, and it equals 3 in the monostatic SAR case. $\mathrm{Tr}(\cdot )$ and $|\cdot |$ indicate the trace and determinant of a matrix, respectively. $K\left(L,d\right)$ is a normalization factor, and $\mathsf{\Gamma }(\cdot )$ denotes the gamma function. ${{\displaystyle \sum }}^{}=E\left\{k{k}^{†}\right\}$ is the expectation of the covariance matrix, which is generally obtained by averaging the covariance matrices within a region or segment.

2.2. KummerU Distribution

The Wishart distribution is suitable for homogeneous areas, whereas less effective in modeling heterogeneous areas with rich texture information. The non-Gaussian product model describes $C$ as the product of two independent stochastic variables:

$$C=ZW,$$

(5)

where the strictly positive unit mean scalar variable $Z$ models texture under the assumption that it is common for all polarimetric channels [23]. The matrix variable $W$ models the speckle term which follows the complex Wishart distribution. Then the PDF of $C$ depends on the scalar texture variable $Z$. When the gamma distribution is adopted, the resulting distribution for $C$ is called the K distribution, and the inverse gamma distribution deduces the G⁰ distribution. Fisher distribution is equal to the Mellin convolution of a Gamma distribution by an inverse Gamma distribution [24], and it can fit well within a large range of SAR clutter. Based on the Fisher distribution, Bombrun et al. [24] proposed the KummerU distribution of the MLC covariance matrix, and the PDF is presented as:

$$\begin{gathered} p(C|{{\displaystyle \sum }}^{},L,\xi ,\zeta )=\frac{L^{dL}{\left|C\right|}^{L-d}}{K\left(L,d\right){\left|{{\displaystyle \sum }}^{}\right|}^L}\frac{\mathsf{\Gamma }\left(\xi +\zeta \right)}{\mathsf{\Gamma }\left(\xi \right)\mathsf{\Gamma }\left(\zeta \right)}{\left(\frac{\xi }{\zeta -1}\right)}^{Ld}\mathsf{\Gamma }\left(Ld+\zeta \right) \\ \cdot U\left(Ld+\zeta , Ld-\xi +1,L\cdot \mathrm{Tr}\left({{{\displaystyle \sum }}^{}}^{-1}C\right)\frac{\xi }{\zeta -1}\right), \end{gathered}$$

(6)

where $\xi$ and $\zeta$ are two positive shape parameters which control the behavior of the Fisher PDF between the heavy head and heavy tail distribution. $U\left(\cdot , \cdot , \cdot \right)$ denotes the second kind of confluent hypergeometric KummerU function. The KummerU distribution encompasses the other statistical models as asymptotic cases, such that it converges to the K distribution when $\zeta \to \infty$, to the G⁰ distribution when $\xi \to \infty$, and to the Wishart distribution when both $\xi ,\zeta \to \infty$.

2.3. Parameter Estimation of KummerU Distribution

Since the maximum likelihood estimation of the parameters is not available with closed-form solutions, it is usually replaced by the numerical estimations instead. Anfinsen et al. [35] proposed to use a matrix-variate Mellin transform in the statistical analysis of MLC PolSAR data, and estimated the distribution parameters using the method of matrix log-cumulants (MoMLC). The MoMLC method has relatively simple numerical expression and possesses lower bias and variance compared with other practical alternatives. The $v$-th order matrix log-cumulant of the covariance matrix $C$ under the product model can be derived as:

$${\kappa }_v\left\{C\right\}={\kappa }_v\left\{W\right\}+d^v{\kappa }_v\left\{Z\right\}.$$

(7)

Based on the texture variable log-cumulants of the Fisher distribution, the MoMLC equations of the MLC covariance matrix are provided as:

$${\kappa }_1\left\{C\right\}={\psi }_d^{\left(0\right)}\left(L\right)+\ln \left|{{\displaystyle \sum }}^{}\right|+d\left({\psi }^{\left(0\right)}\left(\xi \right)-{\psi }^{\left(0\right)}\left(\zeta \right)+\ln \left(\frac{\zeta -1}{L\xi }\right)\right),$$

(8)

$${\kappa }_v\left\{C\right\}={\psi }_d^{\left(v-1\right)}\left(L\right)+d^v\left({\psi }^{\left(v-1\right)}\left(\xi \right)+{\left(-1\right)}^v{\psi }^{\left(v-1\right)}\left(\zeta \right)\right),v>1,$$

(9)

where ${\psi }^{\left(v\right)}(\cdot )$ is the polygamma function, ${\psi }_d^{\left(v\right)}\left(L\right)$ is the $v$-th order multivariate polygamma function, as:

$${\psi }_d^{\left(v\right)}\left(L\right)={\displaystyle \sum }_{i=0}^{d-1}{\psi }^{\left(v\right)}\left(L-i\right).$$

(10)

Figure 1 shows the manifolds spanned by the theoretical second order and third order matrix log-cumulants of different distributions, and this is an extension of the univariate log-cumulant diagram [36]. We can see that the Wishart distribution is represented by a point (black circle) in the image. The K and G⁰ distributions are with one texture parameter, and their manifolds are represented by the red curve and blue curve, respectively. The matrix log-cumulants of the KummerU distribution occupies the whole yellow space. In addition, we also calculate the sample matrix log-cumulants based on the covariance matrices of ESAR (Experimental SAR system operated by DLR, hereafter it is called ESAR for short) data, which will be used in the experiments of the paper. In Figure 1, the sample matrix log-cumulants acquired from three different land covers are shown, including vegetation (cyan), forest (green), and an urban area (magenta). We can see their corresponding sample matrix log-cumulants are different in cluster shapes and locations, however most of them can be covered by the KummerU distribution. According to Equation (9), a pair of the texture parameter $\left(\xi ,\zeta \right)$ corresponds to a point $\left({\kappa }_3,{\kappa }_2\right)$ in the yellow space, and the other way round, the texture parameters can be estimated based on the sample matrix log-cumulants of the PolSAR data.

In practice, the sample matrix log-cumulants $\langle {\kappa }_v\left\{C\right\}\rangle$ is calculated based on the sample matrix log-moments $\langle {\mu }_v\left\{C\right\}\rangle$ given the relationship between them. The equations are presented as:

$${\kappa }_v\left\{C\right\}={\mu }_v\left\{C\right\}-{\displaystyle \sum }_{i=1}^{v-1}\left(\begin{array}{c}v-1\\ i-1\end{array}\right){\kappa }_i\left\{C\right\}{\mu }_{v-i}\left\{C\right\},$$

(11)

$$\langle {\mu }_v\left\{C\right\}\rangle =\frac{1}{N}{\displaystyle \sum }_{i=1}^N{\left(\ln \left|C_i\right|\right)}^v.$$

(12)

Using the second order and third order matrix log-cumulants, a squared Mahalanobis distance can be defined to estimate the texture parameter $\mathsf{\theta }=\left(\xi ,\zeta \right)$ of the KummerU distribution. The distance is calculated as:

$$d_M^2={\left(\langle \mathit{\kappa }\rangle -\mathit{\kappa }\right)}^T{K}^{-1}\left(\langle \mathit{\kappa }\rangle -\mathit{\kappa }\right),$$

(13)

where $\langle \mathit{\kappa }\rangle$ denotes the sample matrix log-cumulant vector $\langle \mathit{\kappa }\rangle ={\left[\langle {\kappa }_2\rangle ,\langle {\kappa }_3\rangle \right]}^T$, and $\mathit{\kappa }$ denotes its mean $\mathit{\kappa }={\left[{\kappa }_2,{\kappa }_3\right]}^T$. The covariance matrix $K$ is defined as [37]:

$$K=\left[\begin{array}{cc}{\kappa }_4+2{\kappa }_2^2& {\kappa }_5+6{\kappa }_2{\kappa }_3\\ {\kappa }_5+6{\kappa }_2{\kappa }_3& {\kappa }_6+9{\kappa }_2{\kappa }_4+9{\kappa }_3^2+6{\kappa }_2^3\end{array}\right].$$

(14)

In Equation (13), $\langle \mathit{\kappa }\rangle$ can be calculated based on the collected PolSAR data samples, $\mathit{\kappa }$ and $K$ depend on the texture parameter $\mathsf{\theta }$ according to Equations (9) and (14). Therefore, the parameter estimation can be performed through the following minimization process as:

$$\widehat{\mathsf{\theta }}=\arg \left\{\underset{\mathsf{\theta }}{\min }\left\{d_M^2\right\}\right\}.$$

(15)

3. Methodology

In this paper, the superpixels are adopted as the basic operation units of the proposed segmentation method. A superpixel is defined as a homogeneous region, which preserves the local characteristics and image boundaries well. For optical images, many superpixel segmentation methods have been proposed, whereas they cannot be directly used for PolSAR data due to the inherent speckle noise. Our recent work [38] proposed a novel superpixel generation method for PolSAR images. This method first calculates the distances between neighboring pixels, and then generates the superpixels based on the entropy rate framework [39], which is implemented through a global optimization way. The distance between neighboring pixels is measured by calculating the dissimilarity between corresponding small regions, therefore the speckle noise effect can be relieved to a large extent. In addition, a directional span-driven adaptive (DSDA) region was proposed such that it contains homogeneous samples only, then the distance measure is more reliable and accurate than the distance which is based on the classic rectangle regions. As to the entropy rate framework, its objective function contains two terms: the entropy rate term, which favors formation of compact and homogeneous regions, and the balancing term, which advocates regions with similar size. This superpixel segmentation method can obtain satisfactory results in both homogeneous and heterogeneous areas, and is applied to initialize the superpixel segments in this paper. Please refer to [38] for more details about the adopted superpixel segmentation method.

For a given PolSAR image, the initial superpixels are denoted by $\left\{R_1,R_2,\cdots ,R_n\right\}$, and $n$ means the number of the superpixels. Each region $R_i\left(i=1,2,\cdots ,n\right)$ is relatively homogeneous and disjoint from others. Then, the image can be represented by a region adjacency graph (RAG) $G=\left(V,A\right)$, where $V$ is the set of superpixels as vertices of the graph and $A$ is the set of arcs connecting spatially adjacent regions. An arc $a_{ij}\in A$ means the shared boundary between two neighboring superpixels $R_i$ and $R_j$.

The hierarchical segmentation [30,40] is a classical iterative merging algorithm, in which the two regions meeting a certain criterion are iteratively merged. From the statistical view, merging two regions yields a decrease of the energy function (or log-likelihood function), therefore the two adjacent regions which can minimize the energy loss should be merged in each iteration step [29]. Let $R_{ij}=R_i{{\displaystyle \cup }}^{}{R}_j$ be the resulted region after merging $R_i$ and $R_j$, then the energy loss caused by this merging can be derived as:

$$\Delta E=E\left(R_i\right)+E\left(R_j\right)-E\left(R_{ij}\right).$$

(16)

The energy $E(\cdot )$ is calculated based on the likelihood function of the MLC covariance matrix, as:

$$E\left(R\right)={\displaystyle \sum }_{s\in R}\ln \left(p\left(C_s|{\Theta }_R\right)\right),$$

(17)

where $\mathrm{s}$ represents a pixel, and ${\Theta }_R$ denotes the parameters of region $R$.

As discussed in the previous section, the MLC covariance matrix follows the complex Wishart distribution in homogeneous areas, whereas the KummerU distribution can better characterize the clutter in heterogeneous areas with rich texture information. However, the PDF calculation based on the KummerU distribution is very time-consuming since it involves the KummerU function and the estimation of the texture parameters. In order to segment the PolSAR image with good accuracy and high efficiency, this paper proposes a novel framework consisting of two merging stages, i.e., the WMS and the KUMS. In brief, the WMS focuses on merging the superpixels in homogeneous areas at a high speed. Based on the resultant segments, the KUMS follows the iterative merging procedure, and generates the final segmentation result with high accuracy. The details of the proposed two-stage merging strategy are explained as follows.

3.1. Wishart-Merging Stage

Under the assumption of the Wishart distribution, the energy function can be derived based on the PDF of covariance matrix in Equation (3). Eliminating the unrelated constant parts gives the total energy of region $R$ as:

$$\begin{array}{ll}\hspace{1em}{E}^w\left(R\right)& ={\displaystyle \sum _{s\in R}}\{\left(L-d\right)\ln \left|C_s\right|-L\cdot \mathrm{Tr}\left({{{\displaystyle \sum }}^{}}^{-1}{C}_s\right)-L\cdot \ln \left|{{\displaystyle \sum }}^{}\right|\}\\ & =\left(L-d\right){\displaystyle \sum _{s\in R}}\ln \left|C_s\right|-L\cdot \mathrm{Tr}\left({{{\displaystyle \sum }}^{}}^{-1}{\displaystyle \sum _{s\in R}}{C}_s\right)-L\cdot {\left|R\right|}_{\mathrm{c}}\ln \left|{{\displaystyle \sum }}^{}\right|,\end{array}$$

(18)

where ${\left|R\right|}_c$ denotes the cardinality of region $R$. Since the distribution parameter ${{\displaystyle \sum }}^{}$ is estimated as the average of the covariance matrices within $R$, the term ${{{\displaystyle \sum }}^{}}^{-1}{\displaystyle \sum }_{s\in R}{\mathbf{C}}_s$ equals ${\left|R\right|}_c\mathbf{I}$ in which $\mathbf{I}$ is the unit matrix, and the energy function can be simplified as:

$$E^w\left(R\right)=\left(L-d\right){\displaystyle \sum }_{s\in R}\ln \left|C_s\right|-dL{\left|R\right|}_{\mathrm{c}}-L{\left|R\right|}_{\mathrm{c}}\ln \left|{{\displaystyle \sum }}^{}\right|.$$

(19)

For the two adjacent superpixels $R_i$ and $R_j$, the energy loss in Equation (16) can be obtained accordingly. Noting that the first two terms in $E^w\left(R_i\right)$, $E^w\left(R_j\right)$, and $E^w\left(R_{ij}\right)$ will be offset, then removing the constant factor $L$ yields the final energy loss function as:

$$\Delta E_{ij}^w={\left|R_{ij}\right|}_{\mathrm{c}}\ln \left|{{{\displaystyle \sum }}^{}}_{ij}\right|-{\left|R_i\right|}_{\mathrm{c}}\ln \left|{{{\displaystyle \sum }}^{}}_i\right|-{\left|R_j\right|}_{\mathrm{c}}\ln \left|{{{\displaystyle \sum }}^{}}_j\right|.$$

(20)

When ${{{\displaystyle \sum }}^{}}_i={{{\displaystyle \sum }}^{}}_j$, the energy loss $\Delta E_{ij}^w$ has a minimum value, zero, and it becomes larger if they are from different land covers. So the energy loss function can also be regarded as a distance measure between two regions, as used in [15,20]. In addition, the edge penalty [41] is incorporated with the energy loss function to strengthen the merging criterion. The edge penalty function between $R_i$ and $R_j$, and the obtained merging criterion are expressed as:

$$E_{P_{ij}}={\displaystyle \sum }_{s\in a_{ij}}\{1-e^{-{(V_s/K)}^2}\},$$

(21)

$${\mathrm{SC}}_{ij}=\Delta E_{ij}^w+\beta \cdot E_{P_{ij}},$$

(22)

where $\beta$ controls the relative weight of $E_{P_{ij}}$ over $\Delta E_{ij}^w$, $K$ is a parameter determining the penalty strength, and $V_s$ denotes the edge strength value of pixel $s$, which can be obtained by the edge detection method in [42]. The edge strength of each pixel is represented by the dissimilarity between both sides of the pixel under study, and the strength value is normalized to $\left[0,1\right]$ in this paper. In the case of strong and obvious edges, the edge penalty $E_{P_{ij}}$ is quite significant, thus preventing these two regions from being merged. $K$ is set as 0.3 in the paper, and the selection of the parameter value will be discussed in Section 4.1 in details.

In order to have a visual perception of the edge penalty between neighboring superpixels, an area of ESAR data is selected for representation. Figure 2a shows the Pauli-coded image and the overlaid green lines denote the boundaries of the superpixel segments. Figure 2b presents the edge detection result of this area, and it shows that the edge map is satisfactory in both homogeneous and heterogeneous areas. Figure 2c shows the boundaries of superpixels, and Figure 2d is the edge penalty map in which the value of the superpixel boundaries is set as the edge penalty strength between the corresponding superpixels. We can see that edge penalty in heterogeneous areas is prominent due to the obvious edges. To the contrary, the edge penalty in homogeneous areas is quite weak. Therefore, the edge penalty term can prevent the superpixels in different land covers from being merged, and at the same time, it exerts little effect in homogeneous areas.

Based on the merging criterion in Equation (22), the superpixel candidates to be merged and the merging condition are expressed as follows:

$$\mathrm{D}=\left\{\left(R_i, R_j\right)|{\mathrm{SC}}_{ij}\le \sigma \right\},$$

(23)

where $\sigma$ is the threshold that determines the pairs of adjacent superpixels to be merged in the WMS. The WMS is the first merging stage, and it merges the superpixels without ambiguity, which not only have no obvious edge but also generate small energy loss. The merging criterion $SC$ is computationally efficient. In addition, the set $D$ is determined based on the initial superpixel partition, and the merging criterion $SC$ need not to be updated in the procedure. So the WMS can simultaneously merge the superpixels with high efficiency.

3.2. KummerU-Merging Stage

The KUMS is dedicated to merging the resultant regions from the WMS, and the KummerU distribution is applied to calculate the energy loss from the statistical aspect. After substituting Equation (6) into Equation (17) and eliminating the unrelated constant parts, the KummerU energy $E^u\left(R\right)$ is given by:

$$\begin{gathered} E^u\left(R\right)=\left(L-d\right){\displaystyle \sum }_{s\in R}\ln \left|C_s\right|-L{\left|R\right|}_{\mathrm{c}}\ln \left|{{\displaystyle \sum }}^{}\right|+{\left|R\right|}_{\mathrm{c}}\ln \left\{\frac{\mathsf{\Gamma }\left(\xi +\zeta \right)\mathsf{\Gamma }\left(Ld+\zeta \right)}{\mathsf{\Gamma }\left(\xi \right)\mathsf{\Gamma }\left(\zeta \right)}\right\}+Ld{\left|R\right|}_{\mathrm{c}}\ln \left\{\frac{\xi }{\zeta -1}\right\} \\ +{\displaystyle \sum }_{s\in R}\ln \left\{U\left(Ld+\zeta , Ld-\xi +1,L\cdot \mathrm{Tr}\left({{{\displaystyle \sum }}^{}}^{-1}{C}_s\right)\frac{\xi }{\zeta -1}\right)\right\}. \end{gathered}$$

(24)

The last three terms can be denoted as $E^{\prime }\left(R|{\Theta }_R\right)$ where ${\Theta }_R$ contains ${{\displaystyle \sum }}^{}$ and the shape parameters $\left(\xi ,\zeta \right)$ that can be estimated by the MoMLC method. Based on the estimated parameters, the KummerU energy loss is derived as:

$$\begin{array}{l}\Delta E_{ij}^u={\left|R_{ij}\right|}_{\mathrm{c}}\ln \left|{{{\displaystyle \sum }}^{}}_{ij}\right|-{\left|R_i\right|}_{\mathrm{c}}\ln \left|{{{\displaystyle \sum }}^{}}_i\right|-{\left|R_j\right|}_{\mathrm{c}}\ln \left|{{{\displaystyle \sum }}^{}}_j\right|\\ \hspace{1em}\hspace{1em}\hspace{1em}+\frac{1}{L}\left[E^{\prime }\left(R_i|{\Theta }_{R_i}\right)+E^{\prime }\left(R_j|{\Theta }_{R_j}\right)-E^{\prime }\left(R_{ij}|{\Theta }_{R_{ij}}\right)\right].\end{array}$$

(25)

We can find that the first three terms in Equation (25) are similar to the Wishart energy loss in Equation (20). The rest can be viewed as the correction term introduced by the texture modeling of the PolSAR data [29]. As is known, the merging procedure reduces the number of the segments, meanwhile, the homogeneity of the retained regions should be kept. So the merging criterion essentially enforces the homogeneity of the newly merged region. In the paper, the homogeneity measurement is also applied in the KUMS. The coefficient of variation (CoV) [43,44] can be used as an index of homogeneity, and it is adapted to the multiplicative speckle noise. When the two adjacent regions are distinct from each other, the homogeneity of the merged region will be significantly changed compared to that of the individual regions. Therefore, a factor named homogeneity penalty is proposed in the paper as:

$$F_h=\frac{\left|H_{ij}-\min \left(H_i,H_j\right)\right|}{\left|H_{ij}+\min \left(H_i,H_j\right)\right|},$$

(26)

where $H$ denotes the homogeneity of a region, i.e., the CoV. Based on the KummerU energy loss $\Delta E_{ij}^u$, the edge penalty function $E_{P_{ij}}$ and the homogeneity penalty $F_h$, the new merging criterion in the KUMS is expressed as:

$${\mathrm{SC}}_{ij}=F_h\cdot \left(\Delta E_{ij}^u+\beta \cdot E_{P_{ij}}\right).$$

(27)

The homogeneity penalty $F_h$ is quite small when the two regions $R_i$ and $R_j$ are from the same land cover, making these two regions be merged first. To the contrary, when they are more distinct, the merged region is heterogeneous, and $H_{ij}$ will be obviously different from $H_i$ and $H_j$. Therefore, $F_h$ penalizes more, and the merging of these two regions will be constrained.

The KUMS follows the iterative merging procedure using the merging criterion in Equation (27). In each iteration step, the two regions minimizing the criterion $SC$ are merged, and the criterion $SC$ related to the newly merged region should be updated. If no halt condition is applied, the merging procedure may continue until the image is converged to a single segment, i.e., the number of regions is one. Therefore, the iterative procedure transfers an over-segmented image to an under-segmented one. However, it is difficult to determine the most appropriate partition among the intermediate steps, and this problem is not widely studied in the literature. The L-method [45] is applied in this paper to set the number of regions. The iterative merging steps of the KUMS are listed in the Algorithm 1 as follows.

Algorithm 1: KUMS iterative merging.
Begin
1.	For each pair of adjacent regions, calculate the KummerU energy loss $\Delta E_{ij}^u$, the edge penalty function $E_{P_{ij}}$, and the homogeneity penalty Fh, then obtain the merging criterion SCij, according to Equation (27)
2.	Find the minimum SCij, and merge the corresponding adjacent regions.
3.	Update the region adjacency graph RAG, and re-calculate the merging criterion SCij which is related to the newly merged region, and continue from Step 2.
4.	Determine the appropriate number of the regions based on the L-method, and the final segmentation result is obtained accordingly.
End

3.3. Procedure of the Proposed Method

The integral flowchart of the proposed segmentation method is illustrated in Figure 3. Superpixel segmentation of the PolSAR image is first implemented to obtain the initial segments, followed by the two-stage merging procedure. The WMS makes use of the Wishart energy and the edge penalty to merge the superpixels without ambiguity. It can efficiently merge the superpixels based on the determined merging sequence. The KUMS iteratively merges the remaining regions (i.e., the regions with ambiguity) by applying the doubly flexible KummerU distribution. The WMS accelerates the process of the proposed segmentation method, and the KUMS is used to improve the accuracy of the final segmentation result.

4. Experimental Results and Analysis

In this section, experimental results of two real PolSAR data sets are presented to show the performance of the proposed segmentation method. The classical hierarchical segmentation schemes, i.e., iterative merging based on the Wishart criterion (hereafter refer to as IM-Wishart) and iterative merging based on the KummerU criterion (IM-KummerU) are also implemented for comparison.

4.1. Datasets Description and Parameter Settings

The first experimental data set is a subset of the multi-look PolSAR image acquired by the ESAR L-band system. The study area is located in Oberpfaffenhofen, Germany, and the region size is $510\times 530$ pixels. For the purpose of numerical evaluation and comparison of different methods, the ground truth map is obtained through segmenting different land covers manually. Figure 4a shows the Pauli image, in which the green lines depict the ground truth segmentation. We can see that there are many different kinds of land covers in this area, including man-made buildings, forests, farm lands, and so on. The second PolSAR image is an agricultural area from the EMISAR (the SAR system operated by the Electromagnetics Institute of the Technical University of Denmark, hereafter it is called EMISAR for short) quad-pol acquisition over Foulum, Denmark. Figure 4b shows the Pauli image, and its size is $600\times 600$ pixels. Similarly, this region is covered by farm lands, as well as forest and buildings. Therefore, these two regions cover both homogeneous areas and heterogeneous areas, containing the information of varied textures.

The parameters of the proposed segmentation method are set as follows. The weighting factor $\beta$ of the edge penalty function is set to 5 according to [16]. The threshold $\sigma$ in Equation (23) is adaptively determined so that a certain proportion of the superpixels can be merged in the WMS. In this paper, 50% of the initial superpixels are efficiently merged in the first stage. The experiments show that these settings work well and can achieve satisfactory results. In the KUMS, the only parameter is the region number of the final segmentation result, and it is set using the L-method, as introduced in Section 3.2.

The parameter $K$ of the edge penalty function in Equation (21) was set experimentally using numerical evaluation. The edge penalty is used to strengthen the merging criterion, and to better preserve the true edges. Therefore, we adopt the measurement named boundary recall which indicates how well the true edges are kept. Based on the partitions after WMS, the recall value can be obtained by calculating the fraction of boundaries that are true from the overall ground truth boundaries. Figure 5a presents the penalty strength of a single pixel, with the parameter $K$ varying from 0.1 to 0.5. In Figure 5b, the blue curve shows the obtained recall as the $K$ parameter changes. The red line denotes the recall value without edge penalty, and it is used as a baseline. From Figure 5a, we can see that a small $K$ leads to excessive penalty of small edge value. Then the merging among the regions is perturbed, and some regions would be merged incorrectly. In such case, the recall is not satisfactory, even worse than that without edge penalty, as shown in Figure 5b. However, when $K$ is too big, the penalty of the edges with medium and large edge strength is inadequate, and the recall drops. To make a compromise, $K$ is set as 0.3, which also achieves the highest recall value in the experiments, as shown in Figure 5b.

4.2. Segmentation Results of ESAR Data

Figure 2a gives the initial superpixel segmentation result of the ESAR data, and the number of the superpixels is 706. The proposed method firstly merges the adjacent superpixels without ambiguity based on the Wishart energy loss and the edge penalty. Figure 6 shows the intermediate segmentation result after the WMS, and the number of the segments reduces to 353. It can be clearly noticed that a majority of superpixels within the homogeneous areas have been merged in this stage, and the remaining superpixels are located in the residential areas or along the boundaries between different land covers. Therefore, the WMS can effectively merge the neighboring superpixels without ambiguity, leaving the regions in heterogeneous areas to the KUMS.

Before presenting the final segmentation results of different merging strategies, the optimal region number is obtained at first. In the iterative merging procedure, the total energy of each intermediate partition can be easily calculated. Figure 7a,b illustrate the energy function of the last 350 merging steps, using IM-Wishart method and IM-KummerU method, respectively. With the decrease of the region number, the energy function changes slightly, and then drops drastically in the end. The knee of the energy curve corresponds to the optimal number of the regions. The small area around the knee is selected and enlarged for careful inspection. Using the L-method, the optimal region number based on the Wishart criterion is obtained as 22, and it is consistent with the knee in Figure 7a, as marked with the red circle. Similarly, the estimated region number based on the KummerU criterion is 37. The optimal region number denotes the point after which the adjacent regions are obviously different and they should not be merged any longer. Considering the estimated region number of the KummerU criterion is larger than that of the Wishart criterion, it can be inferred that the KummerU criterion can better distinguish the difference between the adjacent regions. In the following segmentation experiments, for a fair comparison, the region numbers of all the three methods are set uniformly, i.e., 37 for ESAR dataset.

Given the optimal region number, the proposed method iteratively merges the adjacent similar regions based on the intermediate partition from the WMS. Whereas IM-Wishart and IM-KummerU execute the iterative merging based on the initial superpixel partition. Figure 8 presents the final segmentation results of these three methods, with the first, second, and the third column corresponding to IM-Wishart, IM-KummerU, and the proposed method, respectively. Figure 8a–c show the final segmentation maps, where the green lines superimposed onto the Pauli image depict the boundaries of the segmented regions. On the whole, the results of these three methods are satisfactory, since most of the superpixels in homogeneous areas are merged and the obviously different land covers are retained. Figure 8d–f are the representation maps of the segmented results, in which the covariance matrices in each region are averaged. The segments in all the three maps are similar in shape, however, we can also observe some discrepancies that arise from the different merging strategies of the methods. By comparison, the methods using the KummerU criterion, i.e., IM-KummerU and the proposed method, can better discriminate the different land covers and prevent them from being merged. As marked with a red square in the top right corner, area A1 of Figure 8d is composed of three kinds of land covers. However, area A1 of Figure 8e,f is more consistent with the reality. Area A2 also contains several land cover types, in which a region colored dark blue is located in the right side, as shown in Figure 8e,f. However, this region is merged together with its neighboring segment in Figure 8d. For the IM-Wishart method, only the determinant of the center covariance matrix is used, as implied in Equation (20), and both the structure of the covariance matrix and the texture of the clutter are ignored [29]. By contrast, the KummerU criterion takes the texture information into account, and employs the full information of the covariance matrices. Therefore, it can better distinguish between regions that generate little Wishart energy loss and have different textural properties.

In addition to the KummerU criterion, the KUMS of the proposed method also applies the edge penalty function and the homogeneity penalty to guide the merging process. Since the homogeneity penalty term advocates merging the homogeneous regions and it penalizes more when the two adjacent regions are more distinct, then the different land covers can be better preserved. As shown in Figure 8f, the object in the middle of area A2 is preserved, whereas it is merged into the forest in Figure 8d,e. Area B1 and B2 are two complex residential areas where several kinds of land covers exist. Figure 8g–i are the enlarged representation maps of B1 (left side) and B2 (right side). Compared with Figure 8g,h, we can see that Figure 8i contains more details of the land covers, especially the man-made buildings which are colored pink, as marked using red ellipses. These experimental results visually demonstrate the effectiveness of the proposed method.

In order to numerically evaluate the segmentation performance, the precision-recall framework [32,46] is adopted in the paper, which calculates the precision and recall measurements based on the ground-truth maps. Precision is the fraction of boundaries that are true from the overall detected boundaries, and recall is previously introduced in Section 4.1. In addition, these two measures can be combined together as a single F-measure, and its formula is given by:

$$F=\frac{2\cdot \mathrm{Precision}\cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.$$

(28)

Figure 9 presents the precision-recall curves of different methods, and the gray lines represent the Iso-F curves. In the procedure of iterative merging, the precision and recall values can be obtained based on the intermediate partitions. For most of the early merging steps, the merged regions are within homogeneous areas, then the overall detected boundaries decrease, whereas the detected true boundaries remain the same, therefore the recall changes slightly and the precision increases gradually. When the merged regions are from different land covers, the true boundaries between them vanish and cannot be detected, then the recall reduces accordingly. This phenomenon is more obvious with regard to the last merging steps, which cause serious under-segmentations. It can be seen that the proposed method achieves higher precision and recall values than the other two methods. This is because the KUMS applies the edge penalty and homogeneity penalty in the merging criterion, in addition to the KummerU energy term. Then the different land covers can be better preserved in the merging procedure, and more details and true boundaries can be kept. The precision-recall curves of different methods are consistent with the segmentation results in Figure 8. The numerical evaluation indices of the segmentation results are listed in

Table 1. Numerical evaluation measures of the segmentation results using different methods.

		IM-Wishart	IM-KummerU	The Proposed Method
ESAR	Precision	0.772	0.794	0.8
	Recall	0.758	0.788	0.8
	F-measure	0.765	0.791	0.8
EMISAR	Precision	0.768	0.797	0.81
	Recall	0.76	0.787	0.804
	F-measure	0.764	0.792	0.807

, including precision, recall and F-measure. The maximum F-measure of the proposed method is higher than that of IM-Wishart by 0.035 and IM-KummerU by 0.009. In addition, the proposed method can quickly merge the superpixels without ambiguity through the WMS, therefore accelerates the following iterative merging procedure to a great extent. The computational costs of these three methods will be provided and compared in the end of Section 4.

4.3. Segmentation Results of EMISAR Data

Figure 10a shows the superpixel partition of the EMISAR data, with the number of the superpixels as 1483. Figure 10b presents the intermediate segmentation result obtained from the WMS, which simultaneously merges the adjacent superpixels without ambiguity. The number of segments after merging is 742. By comparing the segmentation results in Figure 10a,b, we can find that many superpixels in the farm lands are merged, and the superpixels in heterogeneous areas are not affected in the WMS.

Using the L-method, the optimal region number of the EMISAR data is obtained as 89. Figure 11 shows the final segmentation maps and the representation maps of IM-Wishart, IM-KummerU, and the proposed method. From the whole perspective, the results of these three methods are quite similar. The superpixels or regions in the individual farm lands are merged, and most of the farm lands can be separated from each other. Even the isolation strips between neighboring farm lands are preserved. However, through careful comparison, it can be found that IM-KummerU and the proposed method can obtain better segmentation results. In Figure 11d–f, area A1 and A2 represent a forest and a residential area, respectively. We can see that these two areas in Figure 11e,f contain more details and are more consistent with the reality than that of Figure 11d, and it shows the flexibility of the KummerU distribution in modelling the heterogeneous and textured areas. Moreover, the proposed method can better preserve the details compared to the IM-KummerU. From the yellow ellipses in Figure 11d–f, we can see that some small regions are maintained by the proposed method, whereas being merged with the neighboring segments by IM-Wishart and IM-KummerU. The numerical evaluation of the segmentation results is also performed using the ground truth data. Since the precision-recall curves are similar to that in Figure 9, they are omitted here. Table 1 also gives the quantitative evaluation measures of different methods on EMISAR data, demonstrating the effectiveness of the proposed method.

All the experiments are implemented on a desktop with an Intel Core i7-4790 CPU of 3.6 GHz and 32 GB memory. The algorithms are programmed using C and Matlab language.

Table 2. Execution time comparison of different methods (in seconds).

	IM-Wishart	IM-KummerU	The Proposed Method
			WMS	KUMS	Total
ESAR	337.5	703.8	2.7	256.2	258.9
EMISAR	453.4	914.6	3.5	371.8	375.3

lists the time costs of different methods. The efficiency of the segmentation methods depends not only on the statistical model that adopted, but also on the segmentation strategy. For each iterative merging step, it needs to update the RAG and the information related to the newly merged region. Most importantly, the IM-KummerU method involves estimating the texture parameters and calculating the complex KummerU function, therefore it takes quite long time to obtain the final segmentation result. As to the proposed method, the WMS is first carried out to simultaneously merge the superpixels without ambiguity. The merging criterion of the WMS is simple to calculate, and the merging sequence is deterministic, therefore the WMS is highly efficient in processing. It takes only 3–4 seconds for the two data sets. The iterative merging of the KUMS is operated on the resultant regions from the WMS, then the time cost can be significantly reduced compared to iterative merging from the initial partition. Using the two-stage merging scheme, the total time cost of the proposed method is much shorter than that of IM-KummerU.

5. Conclusions

In this paper, a two-stage merging algorithm is proposed for PolSAR image segmentation by adopting the superpixels as the operation units. This method applies different criterions to separately merge the regions without ambiguity and the regions with ambiguity. Based on the initial superpixel partition, the WMS applies the Wishart energy loss and the edge penalty to simultaneously merge the regions without ambiguity, which are located in homogeneous areas. Then, more efforts are made in the KUMS to accurately merge the regions with ambiguity. Considering that the doubly flexible KummerU distribution can better characterize the clutter in heterogeneous areas, the KummerU energy loss is adopted to distinguish the regions with different textural information. In addition, the edge penalty and the proposed homogeneity penalty are applied so that the regions from distinct land covers will be penalized and preserved from being merged. In the proposed method, the WMS is used to improve the efficiency whereas the KUMS can increase the accuracy of the segmentation results.

Two real PolSAR datasets acquired by ESAR and EMISAR systems are used to demonstrate the effectiveness of the proposed method, using both visual presentation and numerical evaluation. Compared with the classical iterative merging methods, the proposed two-stage merging strategy can effectively improve the computation efficiency and segmentation accuracy. Since the KummerU distribution is used to characterize the clutter in heterogeneous areas, it can be inferred that the proposed method can obtain more accurate results than the traditional methods. In addition, the proposed method adopts the superpixels as the operation units, it is thus less affected by the speckle noise and can obtain smoother and more visually pleasing results compared to the pixel-based methods.

However, as a superpixel-based segmentation method, the proposed framework heavily relies on the initial superpixel generation. Therefore, unsatisfactory segmentation results can be obtained when the initial superpixels are not well established. In addition, the estimation accuracy of the textural parameters can also be influenced if the samples within a superpixel is too limited. In our further work, more efforts will be made to resolve these issues.