Monitoring Urban Change in Conflict from the Perspective of Optical and SAR Satellites: The Case of Mariupol, a City in the Conflict between RUS and UKR

Abstract

Modern armed conflicts can cause serious humanitarian disasters, and remote sensing technology is critical in monitoring war crimes and assessing post-war damage. In this study, a constrained energy minimization algorithm incorporating the feature bands (IFB-CEM) is designed to detect urban burning areas in optical images. Due to the difficulty of obtaining the ground survey data of the battlefield, the dual-polarization normalized coherence index (DPNCI) is designed based on the multi-temporal synthetic aperture radar (SAR) image, and the quantitative inversion and evaluation of the destruction of urban architecture are combined with the public images on the Internet. The results show that the burning area is widely distributed in the armed conflict region, and the distribution is most concentrated around the Azovstal steel and iron works. The burning area reached its peak around 22 March, and its change is consistent with the conflict process in time and space. About 79.2% of the buildings in the city were severely damaged or completely destroyed, and there was a significant correlation with burning exposure. The results of this study show that publicly available medium-resolution remote sensing data and Internet information have the ability to respond quickly to the damage assessment of armed conflict and can provide preliminary reference information for dealing with humanitarian disasters.

1. Introduction

Modern armed conflicts will have a huge impact on the safety of human life and property as well as the social and natural environment [1]. Since the 21st century, dozens of armed conflicts have broken out in the world, including the war in Afghanistan, the war in Iraq, the civil war in Syria, the civil war in Libya and the Russia–Ukraine conflict. These conflicts not only changed the social, economic and political patterns of a country but also subtly affected the trend of human civilization. The direct impact of armed conflict is the loss of human lives and the destruction of infrastructure. What is more serious is that, due to the lack of access to infrastructure and a safe living environment, conflict indirectly causes mass migration of people and eventually turns into a series of serious humanitarian disasters [2].

Unlike crises caused by natural disasters, the deterioration of regional security during armed conflicts makes it difficult for international observers to conduct field investigations into conflict areas and assess disaster losses. Low-altitude aircraft are also unable to operate due to airspace security problems. Therefore, how to quickly obtain accurate temporal and spatial dynamic information on humanitarian disasters, assess losses in disaster areas and then guide the international community to mediate conflicts and carry out humanitarian aid and post-disaster reconstruction has become an urgent problem for the international community to solve in the face of armed conflict crisis.

The development of earth observation technology provides a better choice to solve this problem. Remote sensing images can quickly obtain a wide range of surface information and effectively detect the occurrence of malignant events and ground object damage in humanitarian disaster areas. Therefore, some research institutions or researchers, including the United Nations Institute for Training and Research (UNITAR), have conducted humanitarian disaster monitoring and damage assessment in Iraq, Yemen, Syria and other countries based on different types of remote sensing images [3,4,5,6].

In high-resolution optical remote sensing images (such as Worldview, Quickbird, GeoEye), individual infrastructure damage and the occurrence of small crowd gatherings can be detected. Marx et al. used Plant, a high-resolution satellite constellation with a spatial resolution of 3 m, to detect the potentially burned villages [7] in the ethnic cleansing campaign of Rohingya villages in northern Rakhine State by the Myanmar military in 2017 [8]. UNITAR evaluated the level of building damage in some areas during the conflict between Russia and Ukraine through visual interpretation based on the Worldview satellite with a resolution of 0.5 m [9]. However, the high cost and small imaging range of high-resolution optical remote sensing images generally limits their application in large-scale armed conflicts. Medium-resolution optical remote sensing images (such as Landsat series, Sentinel-2 series and MODIS) have been widely used in large-scale armed conflict monitoring due to their stable imaging quality, large observation perspective and short revisit period. The main work is reflected in the detection of thermal anomaly areas in conflicts and the detection of regional surface changes. For example, MODIS thermal anomaly detection data products are applied to reveal the potential relationship between fire growth and violent conflicts in Darfur, Sudan, proving that objective information on some violent conflicts may be provided based on the global daily fire detection situation [10]. Landsat ETM+ was used to detect changes in the Darfur region of Sudan, and it was pointed out that the cause of changes might be related to burning by comparing the albedo data of the adjacent two years [11]. Marx modeled the expected daily surface reflectance of each urban pixel in Aleppo and Damascus, Syria, based on the stability and radiative consistency of the Landsat series of optical satellite constellations, combined with their historical archive data, and evaluated the urban building damage in Syria during the conflict [12]. In addition, nighttime light (NTL) can also reflect the impact of armed conflict on human social life to some extent. In the early stage, the Defense Meteorological Satellite Program’s Operational Linescan System (DMSP-OLS) was often used to monitor major events occurring in human social life [13]. Li et al. applied the change of nighttime lighting area during the Syrian crisis for a long time and concluded that there was a linear relationship between the loss of nighttime light and the number of lost population, and confirmed the effectiveness of NTL in monitoring humanitarian crisis [14]. In recent years, the Visible Infrared Imaging Radiometer Suite (VIIRS) sensor on the National Polar-orbiting Partnership (NPP) satellite has become a new source for monitoring light at night, and it can provide better quality data [15], and these data were used in the Russia–Ukraine conflict in 2022 for war action interpretation, socio-economic assessment and refugee population monitoring [16,17,18].

However, optical sensors are usually restricted by environmental factors such as weather, climate and day and night. Especially during the conflict, smoke from burning ground fuels severely blocks the propagation of visible light. Synthetic aperture radar (SAR) provides a new perspective of observing the ground using a microwave. The advantage of all-sky and all-weather makes up for the deficiency of optical sensors affected by observation conditions [19]. Aimaiti, Y et al. used the change of SAR intensity and texture analysis of optical images to detect the damage level of buildings in the Kiev area during the Russia–Ukraine conflict, achieving a detection accuracy of 58% [20]. Washaya P et al. used Sentinel-1 to monitor the coherent changes of natural and man-made disasters occurring in Syria, Iran and other regions, and combined land use type data and coherent map standard deviation to reveal the changes in building damage at street level [21]. In addition, Interferometric Synthetic Aperture Radar (InSAR) technology also provides a more real-time and repeated continuous observation scheme for post-war urban destruction mapping and damage assessment [22,23].

With the development of the Internet and social networks, more and more researchers have noticed the importance of remote sensing images assisted by different sources and multi-view images for earth observation [24]. Optical remote sensing images can be combined with low-altitude aerial photography images and ground reconnaissance images, which have been successful in monitoring the destruction of cultural relics and monuments in parts of Iraq and Syria during the ISIS occupation [25]. The combination of time series NTL and social media big data also has great potential in monitoring and understanding crisis development and refugee flows because both are sensitive indicators of economic and human capital loss, and big data and remote sensing data sets have the potential value of providing classified and timely data for conflicts that lack official statistics [26].

In previous studies, the burning areas in armed conflict are usually based on the band combination of multi-spectral satellites containing infrared bands and observed by human eyes, which is highly subjective and tedious work. Moreover, the detection of ground object damage changes caused by conflicts is usually unable to reach an accurate conclusion due to the lack of effective and timely information references. Therefore, in this study, we designed a semi-automated framework for detecting burning areas in conflict and building damage changes by combining multi-source medium-resolution satellite data with open-source Internet information. In this study, the city of Mariupol, which was severely damaged in the conflict between Russia and Ukraine on 24 February 2022, was taken as an example. The specific work was as follows: (1) The detection of urban burning areas during the battle of Mariupol was carried out by using public Sentinel-2A/B and Landsat-8/9 satellite images; (2) Temporal coherence image of Sentinel-1A satellite was used to deduce the distribution and evolution of damaged buildings in conflicts with Open Street Map, social media data and open data set, and the reliability of the method was tested by coherence changes of burned buildings. (3) This paper comprehensively evaluates the damage situation of urban buildings in Mariupol and discusses the advantages and limitations of using multi-source medium resolution remote sensing satellites in the assessment of armed conflict damage.

2. Study Area and Data

2.1. Study Area

Located in the South of the Donetsk Prefecture and borders the Sea of Azov, it connects the Donbas region to the Crimean Peninsula. Mariupol is the second-largest city in Donetsk Prefecture and the tenth-largest city in Ukraine, the essential sea-land traffic control hub, and an industrialized town for metallurgy, machinery and trade. Mariupol extends from 47°1′26″ N to 47°13′19″ N latitude and from 37°27′41″ E to 37°47′17″ E longitude, covering an area of 244 km². Topographically, it is part of the Sea of Azov lowland, without any mountains or hills, and the elevation of the surrounding farmland is slightly higher than that of the central city. The urban land is mainly concentrated at the mouth of the central Karimius River, with good shipping resources, coupled with its historical coal mining, iron and steel industry heritage, make it a strategic port city. As of the end of 2021, Mariupol had a population of 431,859, with an urban area of 166 km² and a downtown area of 106 km² [27]. Figure 1 shows the geographical location of the study area and optical and SAR images of the corresponding area.

On 24 February 2022, Russia broke out armed conflict with Ukraine on the grounds of “demilitarization and de-Nazification” [28,29,30]. Because of its special geographical location and important industrial resources, Mariupol became a battlefield in the conflict between Russia and Ukraine. During the conflict, not only military facilities were severely destroyed, but also civilian infrastructure, including urban residential areas, commercial centers and religious sites. As of 16 May 2022, the entire territory of Mariupol is under the control of Russian forces. Almost every building in Mariupol has been destroyed, and a large number of people have been killed and injured (no specific statistics are available) during the 82-day conflict, making the area the epicenter of a humanitarian disaster [31].

2.2. Data Source

The remote sensing data used in this study are all from freely available public satellite images. The optical image mainly adopts the Sentinel-2A/2B satellite Level-1C images provided by European Space Agency (ESA) [32]. Sentinel-2 is a wide-swath, high-resolution, multi-spectral imaging mission supporting Copernicus Land Monitoring studies, including the monitoring of vegetation, soil and water cover, as well as observation of inland waterways and coastal areas [33]. Each of the satellites carries a multi-spectral instrument (MSI), covering 13 spectral bands in the visible, near-infrared (NIR), and short-wave infrared (SWIR) ranges: four bands at 10 m, six bands at 20 m and three bands at 60 m spatial resolution [34]. The satellite group has a revisit period of 5 days, and the Level-1C product is the atmospheric apparent reflectance product after ortho correction and geometric precision correction.

In order to make up for some of the images that could not be used due to excessive cloud cover and to form a dense time series data stack, we also collected Level-1C products of the Landsat-8/9 satellite from the United States Geological Survey (USGS) [35]. The payload of the satellite group is the Operational Land Imager (OLI) and the thermal infrared sensor (TIRS), which can acquire images in 11 bands from visible to short-wave infrared range, including eight visible and infrared bands with a resolution of 30 m, one panchromatic band with a resolution of 15 m and two thermal infrared bands with a resolution of 100 m [36]. The sensors carried by Landsat-9 have improved radiation accuracy and have more vital ground object detection ability [37]. The satellite group had a revisit period of 8 days.

In addition to collecting all images that were not obscured or slightly obscured by clouds during the conflict period (24 February 2022–16 May 2022), we also collected two images before the conflict and an image after the conflict for comparative analysis. The specific time and type of optical remote sensing images used in this study are shown in Figure 2.

The Sentinel-1A satellite launched by ESA was used to map the change in urban building damage. It is a C-band ($\mathsf{\lambda }=5.6 \mathrm{cm}$) medium-resolution SAR satellite with four imaging modes: Stripmap (SM), Interferometric Wide Swath (IW), Extra-Wide Swath (EW), and Wave (WV), with a revisit period of 12 days [38]. We acquired Sentinel-1A single look complex (SLC) images in IW mode from the Alaska Satellite Facility [39] from 4 February to 23 May 2022. These images are all descending orbit images with both VH and VV polarization, and the spatial resolution is 5 m × 20 m. The temporal and spatial baselines between Sentinel-1A image pairs adopted in the study are shown in Figure 3; the positive and negative of the spatial baseline represent the difference between the baseline direction and the defined positive direction.

The vector data of buildings, land use types and roads used in the experiment came from the Open Street Map (OSM) platform [40]. These data contributed by platform registrants have the characteristics of inconsistent timeliness and complete alignment between the vector boundary of buildings and ground objects in satellite images. However, we tend to pay more attention to macro-scale damage and its change trend in large-scale armed conflicts. In medium-resolution images, a slight deviation of the building vector will not significantly impact the evaluation results, so we allow such errors. In addition, we have collected a large number of live photographs of the damaged buildings in Mariupol on the Internet platform (these images are from the shared dataset of high-resolution commercial remote sensing satellites [41] and aerial photography from drones, as well as from the ground, as shown in Figure 4) [42,43], and determine its spatial location and damaged degree, and finally, as samples to evaluate the damaged degree of whole urban buildings.

3. Method

The urban change detection framework in conflict proposed in this paper is shown in Figure 5, including four parts: optical image preprocessing, SAR image preprocessing, interest information extraction, and urban damaged change detection and assessment.

First of all, Sen2Cor, SNAP and ENVI were used to preprocess the collected optical satellite images to form a time-series optical image stack. By observing the spectral response value of the urban burning area, the normalized difference fire index (NDFI) was designed and stacked into the original images as the feature band to reconstruct the spectral dimension of the original images. Then, the CEM algorithm was used to quickly extract the burning area in the reconstructed image and accurately identify the urban burning area with a small amount of manual intervention.

SARscape platform was used to correct Sentinel-1A position with precision orbit parameter. After calculating the coherence of SAR image pairs, the coherence map was geocoded by the 1-radian digital elevation model (DEM) of the Shuttle Radar Topography Mission (SRTM). The Dual polarization normalization coherence index (DPNCI) is designed as an index to evaluate the damaged degree of urban buildings, and its sensitivity to damaged buildings is tested by calculating the change rate of DPNCI of burned buildings. Finally, the spatial location and damage degree of buildings in the photos collected on the Internet are marked one by one, and the optimal threshold for distinguishing different damaged degrees of buildings is determined by combining the OSM data so as to form the analysis and evaluation of the temporal and spatial changes of damaged buildings in Mariupol.

3.1. Preprocessing

In order to reduce the influence of the atmosphere on the electromagnetic wave, Sen2Cor was used to conduct atmospheric correction for Sentinel-2A/B Level-1C products to get Level-2A products, then multi-spectral images with a spatial resolution of 10 m were obtained by using SNAP to resampling them and format conversion. At the same time, multispectral images (30 m resolution) and panchromatic images (15 m resolution) after radiation calibration and atmospheric correction of Landsat-8/9 were fused to obtain multispectral images with 15 m resolution. Finally, adopt the same boundary vector to cropped images to obtain the time series optical images.

Used SARscape, the Sentinel-1A image from 4 February 2022, was taken as the master image and the remaining images as the slave images. The dual polarization coherence maps between all image pairs were calculated. After geocoding, image cropping, and resampling, the SAR coherence maps (resolution of 5 m) were obtained, which is consistent with the optical images range.

3.2. Constrained Energy Minimization Algorithm Incorporating the Feature Bands

A $m\times n$ multi-spectral image $S$ with $r$ bands can be expressed as

$$S=[s_1,s_2,s_3,\cdots ,s_{mn}]$$

(1)

where $s_i$ is the spectral vector of the pixel $i$,$s_i=[s_i{}_1,s_i{}_2,s_i{}_3,\cdots ,s_{ir}]$.

To highlight the information on the burning area, we normalized the difference between the SWIR1 band and SWIR2 band in Sentinel-2A/B and Landsat-8/9 images and the NIR band, respectively, to construct two kinds of normalized difference fire indexes ($NDF{I}_i$) [44,45]. The reason for choosing the near-infrared band is that there is usually smoke when the burning event occurs in the conflict zone. Longer wavelength electromagnetic waves can reduce the influence of scattering to a certain extent.

$$NDF{I}_1=\frac{{\rho }_{SWI{R}_1}-{\rho }_{NIR}}{{\rho }_{SWIR1}+{\rho }_{NIR}}$$

(2)

$$NDF{I}_2=\frac{{\rho }_{SWI{R}_2}-{\rho }_{NIR}}{{\rho }_{SWI{R}_2}+{\rho }_{NIR}}$$

(3)

where ${\rho }_{NIR}$, ${\rho }_{SWI{R}_1}$ and ${\rho }_{SWI{R}_2}$ correspond to the bands whose central wavelength is about 0.865 $\mathsf{\mu }\mathrm{m}$, 1.610 $\mathsf{\mu }\mathrm{m}$ and 2.200 $\mathsf{\mu }\mathrm{m}$ in Sentinel-2A/B and Landsat-8/9 images, respectively.

$NDF{I}_i$ was inserted into the image as the feature bands ${\lambda }_1$ and ${\lambda }_2$, and after magnifying them 5000 times, a finite impulse response linear filter $W=[w_1,w_2,w_3,\cdots ,w_r,w_{{\lambda }_1},w_{{\lambda }_2}]$ was designed to minimize the average output energy $WR{W}^T$ of the background sample through the filter by using the known target spectral vector $X=[x_1,x_2,x_3,\cdots ,x_r,x_{{\lambda }_1},x_{{\lambda }_2}]$ as the constraint. In this state, the description of constraints can be followed; that is, while ensuring a high response to the target spectral vector, excessive attention to background information can be reduced so as to effectively extract the target. The objective function and constraint equation is shown in formula (4):

$$\left\{\begin{array}{c}\min \left\{WR{W}^T\right\}\\ s.t.X{W}^T=1\end{array}\right.$$

(4)

where $R=(1/\mathrm{mn})S{S}^T$ represents the autocorrelation matrix of $S$.

Under the constraint of $s.t.X{W}^T=1$, the Lagrange multiplier method is used to solve the optimal filter coefficient:

$$W_{\mathrm{CEM}}=\frac{R^{-1}{X}^T}{X{R}^{-1}{X}^T}$$

(5)

where $R$ represents the autocorrelation matrix of $S$, $X$ is known target spectral vector.

Ultimately, $W_{CEM}$ was applied to each pixel in the image to obtain the distribution of the target in the image and realize the detection of the target.

In the feature bands images, the pixel value in the burning area is much larger than that in the non-burning area. Therefore, after the introduction of the feature bands, the response value of the filter $W$ to the burning area is amplified, while the response value of the non-burning area is suppressed, and the two can be better distinguished.

3.3. Dual Polarization Coherence Change Detection

The damaged buildings can be determined by the actual investigation, image interpretation and other methods. The most accurate method is the actual survey of the disaster site because closer observation can better characterize the extent of the damage. However, it is an extremely difficult challenge to assess the extent of damage from a remote sensing perspective, especially from low- and medium-resolution spaceborne platforms. Therefore, the “damaged change detection” in this study emphasizes explicitly the state that the building is determined to collapse or not collapse, though it does not accurately detect slight or moderate damage, such as wall cracks and outer walls being burnt. Nevertheless, from the point of view of emergency disaster assessment, this may not be important because of the need to quickly assess the number and spatial distribution of highly damaged buildings. In the absence of ground survey data and high-resolution optical images, SAR imaging makes it possible to perform this work because of its all-weather and all-weather characteristics and the availability of coherence detection changes.

In SAR images, buildings in different states have completely different scattered structures. Figure 6 illustrates common types of damage and the corresponding SAR intensity images before and after the damage occurred. (a) is a healthy building, usually with strong backscattering due to the presence of dihedral angles, specular reflection occurs on the roof, and the walls block the signal to form shadows. (b) is the damaged building, the roof and wall collapse formed a disorderly distribution so that the SAR echo signal lost the rule, and the backscattering intensity was significantly weakened. (c) is the damaged type of some large warehouses or factories. The top of the building is damaged, but the rest of the structure is relatively intact. The residual walls and ground may form an angular reflector and cause a strong double-reflection effect [46], and the backscattering intensity is enhanced.

Under ideal geometric conditions, the capability of SAR coherence change detection in building damage detection is better than that of intensity change detection [47]. Phase coherence represents the coherence amount of complex phase signals of SAR images with two different simultaneous phases at the same position. The coherence coefficient $\gamma$ is 0 to 1, which can be defined as

$$\gamma =\frac{E〈ab*〉}{\sqrt{E〈aa*〉E〈bb*〉}}$$

(6)

where $a$ and $b$ are the relative complex values of the preceding time and the following time in the interference analysis, $a*$ and $b*$ are complex conjugate values of the images. $E$ is the mathematical expectation.

In the study area, areas with high vegetation coverage, such as farmland and forest land, often exhibit decoherence due to short-term effects, such as wind disturbance, or long-term effects, such as vegetation growth. The sea surface, rivers and other water bodies will also cause decoherence because the water surface waves constantly change. In urban areas, the main features are artificial buildings, and the coherence is usually at a high level under the condition of not being destroyed by external forces.

The damaged buildings in conflicts can be detected more robustly by synthesizing different polarization modes. In this study, The DPNCI was constructed by a weighted combination of the coherence maps in the VV polarization and VH polarization modes. We selected two SAR images before the conflict, with the image on 4 February 2022, as the master image and the image on February 16 as the slave image, and calculated the coherence coefficient within the artificial building land (Red Area in Figure 7, land types include: commercial, construction, education, gas station, factory, military, religious and residential). In Figure 7, (a) and (b) are VV and VH polarization coherence maps.

Prior to the conflict, the mean coherence values for VV polarization and VH polarization were 0.609 and 0.471, respectively. This suggests that the coherence graph calculated using the VV polarization mode is superior and has a more sensitive index [48]. Therefore, VV should be given more weight than VH when designing the index for damaged discrimination. The DPNCI is defined as follows:

$$DPNCI=0.6{\gamma }_{\mathrm{VV}}+0.4{\gamma }_{\mathrm{VH}}$$

(7)

4. Results and Analysis

4.1. Urban Burning Area Detection

To verify the effectiveness of our method, we randomly selected a number of combustion areas and confusing objects (beach and highlighted building tops) on two types of optical images and compared the response values before and after the method improvement. As shown in Figure 8, IFB-CEM can significantly suppress the response values of non-burning areas in Landsat-8/9 and Sentinel-2A/B images and amplify the response values of some pixels in burning areas, which has a more obvious effect in Sentinel-2 images. (These pixels are usually burning centers, and their response is low in the original CEM algorithm due to the saturation of their response at bands 11 and 12 of Sentinel-2A/B’s sensor). In the local figure at the bottom right corner of Figure 8, IFB-CEM (split by horizontal lines) eliminates background interference to a greater extent and preserves target pixels more completely.

We calculated recall, precision and F1 scores under different thresholds by adjusting response thresholds. Recall is defined as the proportion of correctly extracted burning areas to the number of all actual burning areas, and precision is defined as the proportion of correctly extracted burning areas to the number of all extracted burning areas. The F1 score is defined as the harmonic mean of recall and precision. Since the boundary of the combustion area cannot be accurately divided, we believe that the extracted combustion area and Ground Truth overlap degree is more significant than 0.8, which means that the combustion area at this location is successfully extracted. The best response threshold was determined by the highest F1 score.

Table 1. The optimal extraction results before and after the method improvement.

	CEM			IFB-CEM
	Recall	Precision	F1 Score	Recall	Precision	F1 Score
4 March 2021	0.959	0.839	0.895	0.980	0.959	0.969(↑0.074)
13 January 2022	0.982	0.900	0.939	0.964	0.947	0.955(↑0.016)
13 March 2022	0.981	0.912	0.945	0.943	0.963	0.953(↑0.008)
14 March 2022	1.000	0.922	0.959	0.979	0.979	0.979(↑0.020)
19 March 2022	0.972	0.886	0.927	0.956	0.959	0.957(↑0.030)
21 March 2022	1.000	0.877	0.934	0.972	0.986	0.979(↑0.045)
22 March 2022	0.965	0.933	0.949	0.948	0.976	0.962(↑0.013)
24 March 2022	0.973	0.899	0.934	0.978	0.973	0.975(↑0.041)
29 March 2022	0.986	0.907	0.944	0.986	0.986	0.986(↑0.042)
3 April 2022	0.984	0.861	0.919	0.984	0.964	0.974(↑0.055)
7 April 2022	1.000	0.807	0.893	1.000	0.979	0.989(↑0.096)
15 April 2022	0.933	0.667	0.778	0.933	0.933	0.933(↑0.115)
30 April 2022	1.000	0.333	0.500	1.000	0.667	0.800(↑0.300)
1 May 2022	0.833	0.625	0.714	1.000	0.831	0.908(↑0.194)
3 May 2022	1.000	0.667	0.800	0.883	0.883	0.883(↑0.083)
8 May 2022	0.833	0.625	0.714	0.700	0.840	0.764(↑0.050)
9 May 2022	0.667	0.571	0.615	0.533	0.815	0.645(↑0.030)
28 May 2022	0.500	1.000	0.667	0.500	1.000	0.667(--)

shows the best detection effect of CEM and IFB-CEM methods on each scene. Since the IFB-CEM method suppressed the response of non-target areas, the error detection rate was significantly reduced, and the precision was generally significantly improved.

$$Recall=\frac{TP}{TP+FN}$$

(8)

$$Precision=\frac{TP}{TP+FP}$$

(9)

$$F1 Score=\frac{2\cdot Recall\cdot Precision}{Recall+Precision}$$

(10)

where $TP$, $TN$ and $FP$ are the true positive, true negative and false positive, respectively.

Figure 9 shows the temporal and spatial distribution of burning areas during the conflict. Before and at the beginning of the conflict (before 14 March 2022), the thermal anomalies were distributed in the Ilyich Iron and Steel Works in the north of the city and the Azovstal Steel and iron works in the south of the city, indicating that the production and living activities of the residents were proceeding normally. Images from 14 March showed that the thermal anomaly in the factory area disappeared, meaning that industrial production had to stop due to suffering attack and that all the thermal anomalies since then have been ground objects burning due to the conflict. The Russian forces mainly attacked from east, west and north three directions. The burning area of the city increased dramatically in a short time, showing a trend of gradual encirclement and contraction towards the city center. By the end of March, Russian forces had largely completed the siege of the Azovstal steel and iron works, after consolidating their positions in the city center, had launched an offensive against the Ukrainian 36th Marine Brigade stationed at the Ilyich Iron and Steel Works in the north and had captured the city hall of the Kalmiusky district. In early April, Russian forces began advancing southwest from the city center toward the coast in an attempt to encircle and eliminate Ukrainian forces in the port area, and the burning area shifted accordingly. By the end of April, the thermal anomaly area in the urban area was reduced to the lowest level, and the remnants of the Ukrainian army all retreated to the Azovstal steel and iron works. The Russian army had taken full control of the urban area of Mariupol, and the subsequent fighting basically centered on the Azov steel works, so the burning areas were located in and around the plant. Ukrainian forces surrendered on 16 May, and Sentinel-2A satellite imagery on 28 May showed no burning area in or around the Azovstal steel and iron works, bringing the nearly three-month battle for Mariupol to an end.

4.2. Change Detection of Destroyed Buildings

The calculated DPNCI maps of the time series are shown in Figure 10. Pixel brightness represents the stability of the building area in Mariupol. With the occurrence and persistence of conflicts, white pixels gradually decrease, indicating that the stability of urban building areas is disturbed.

The frequency distribution of the DPNCI in the artificial building land is shown in Figure 11. Its distribution gradually changes from the right skew (before the conflict) to the left skew (after the conflict), and the average coherence keeps decreasing. Especially between 12 March and 24 March, the artificial building showed significant decoherence.

To verify the sensitivity of the DPNCI to the detection of damaged buildings, we superimposed the urban burning area extracted in Section 3.1 with the OSM building vectors, screened out the pixels of buildings located in the burning area, and calculated the changes of their DPNCI values. It is reasonable to assume that, in modern armed conflicts, a weapon capable of causing a building to explode and burn would also do great damage to the structure of the building. Therefore, it is reasonable to consider the buildings in the burning area as damaged buildings.

In Figure 12, the different colored line segments represent the average DPNCI value changes of the burned buildings over three different time periods. The blue line showed the largest decrease in coherence between 12 March and 24 March and remained at a low level, while the green and black lines also showed the largest decrease in coherence between 24 March and 5 April, 5 April and 17 April, respectively. It is proved that this method can accurately detect the time node of damaged buildings.

In order to quantitatively describe the damaged degree and spatial and temporal distribution of buildings in Mariupol, we graded the damage degree and geographically located 543 buildings in the collected images and photos. Four damaged levels were divided according to visual interpretation, as shown in

Table 2. Damaged buildings level.

Damaged Degree	Description	Example	DPNCI
Possible damaged or undamaged	The structure of the building is intact, and no obvious cracks or structural changes can be identified from the images.		0.60–1.00
Moderate damage	The structure of the building is relatively intact, with partially damaged to the top or sides and no apparent collapse.		0.45–0.60
Severe damage	The roof and facade of the building were badly hit, with extensive collapse, but parts of the wall structure remained.		0.27–0.45
Destroyed	The building was completely destroyed in a pile of rubble, making it difficult to see intact parts of the walls.		0.00–0.27

. 80% of the buildings were selected to determine the classification threshold, and the remaining 20% were used as verification data. The confusion matrix of inspection results is shown in Figure 13, among which the classification accuracy of destroyed buildings is the highest, reaching 69%, and the overall accuracy is 59.75%.

As shown in Figure 14, buildings in Mariupol have been severely damaged in general, among which 79.2% are severely damaged or destroyed, mainly in Azovstal steel and iron works and the surrounding central city in the south, Ilyich Iron and Steel Works and municipal center in the north, and around the port area in the southwest.

5. Discussion

Our research shows that the public medium-resolution remote sensing image has the ability of stable response and large range observation in detecting the change of urban burning and damaged buildings caused by armed conflict. However, optical satellite imaging is usually affected by clouds, rain, snow and other weather conditions, especially in a battlefield environment filled with smoke, which greatly affects the effectiveness of earth observation. Therefore, the joint observation of optical and SAR images is the key to solving the problem.

The IFB-CEM we designed allows for better detection of burning areas by more effectively separating burning and non-burning area responses. However, it is greatly affected by the preset sample target spectral curve, so it is necessary to choose the pixel of a burning area with moderate energy as far as possible to balance the detection of the potential weak burning area and super strong burning center area.

In addition, the DPNCI of medium-resolution SAR images can better depict the distribution and change trend of damaged buildings in the conflict [48]. Combined with the area and intensity of nighttime lighting in Mariupol [49], the damaged buildings are closely related to nighttime light changes in time and space. Nevertheless, the lack of high resolution will cause a change of coherence not only from the damage to the building structure. For example, the coherence of small buildings and buildings with high urban greening degrees is easily affected by the scattering of electromagnetic waves by other ground objects, which brings uncertain factors to the detection of damaged grades.

At the same time, we should fully realize the important role of public internet data in assisting remote sensing image interpretation, which can quickly provide relatively reliable information before the lack of real statistics. It should be noted that without understanding the actual situation of the study area, it is extremely difficult and time-consuming to locate these data from the Internet on the map. In the experiment, we locate the damaged buildings in the image by looking for iconic ground objects. For this work, perhaps Cross-View Matching technology for Image-Based Ground-to-Aerial Geo-Localization can be efficiently completed [50]. Moreover, the classification of damaged degree also has a certain degree of subjectivity in this study; the accuracy of damage results obtained by inversion of the empirical model obtained by visual interpretation and statistics still has great room for improvement. In fact, the humanitarian disaster survey results released by UNITAR are a valuable reference, and we also note that these data have been used as an assessment basis in relevant studies [20,22]. Figure 15 shows the damaged buildings result of Mariupol interpreted and released by UNITAR according to the worldview-2 image on 14 March 2022 [51]. Regrettably, we did not adopt it because we do not know what standard UNITAR uses to divide the damage degree of buildings, and the worldview-2 image and Sentinel-1 image are not acquired simultaneously, and the time difference may lead to different damage degrees of the same building, especially in the most intense period of conflict.

According to the above discussion and the inversion results of the damage degree of the building, we believe that the overall low accuracy is also due to the time difference between SAR images and acquired images or photos from the Internet.

6. Conclusions

In this study, the city of Mariupol, which was severely affected during the conflict between Russia and Ukraine, was taken as the research area. Landsat-8/9, Sentinel-1/2, and open-source community data on the Internet, which were publicly available, were used to monitor the temporal and spatial changes of urban burning areas and damaged buildings. The IFB-CEM designed by us has achieved excellent results in detecting burning areas. Through statistical analysis of the burning area, the intensity of conflict reached its peak around 22 March 2022. The spatial growth of damaged buildings was positively correlated with the course of the conflict. By the time the Russian military took full control of Mariupol (End of May 2022), almost all buildings in the city had been hit, with the cumulative proportion of destroyed and severely damaged buildings being 79.2%.

Obviously, medium-resolution SAR satellites are not capable of detecting the extent of damage to small buildings, even though they can provide a large-scale view of the distribution and trend of damage. The timing mismatch between social media pictures and remote sensing images is also one of the important reasons that restrict the accuracy of building damage degree inversion. With today’s convenient access to massive data, it is difficult to locate the location of social media image events only by visual interpretation, and a more intelligent and automated process is urgently needed to assist this work. In the future, we will build a more streamlined armed conflict monitoring platform, design more sophisticated damaged building indicators and rating standards based on higher resolution SAR images, give full play to the potential of current open source data and public Internet data in assisting conflict damage detection, expand our detection scope, and carry out fine-grained conflict damage assessment.