Development of the Statistical Errors Raster Toolbox with Six Automated Models for Raster Analysis in GIS Environments

Abstract

The Statistical Errors Raster Toolbox includes models of the most popular error metrics in the interdisciplinary literature, namely, root mean square error (RMSE), normalized root mean square error (NRMSE), mean bias error (MBE), normalized mean bias error (NMBE), mean absolute error (MAE) and normalized mean absolute error (NMAE), for computing the areal errors of any raster file in .tiff format as compared with a reference raster file. The models are applicable to any size of raster files, no matter if no-data pixels are included. The only prerequisites are that the two raster files share the same units, cell size, and projection system. The novelty lies in the fact that, to date, there is no such application in ArcGIS Pro 3/ArcMap 10.8. Therefore, users who work with raster files require external software, plus the relevant expertise. An application on the reference evapotranspiration (ETo) of Peloponnese peninsula (Greece) is presented. MODIS ET products and ETo raster files for empirical methods are employed. The results of the models (for 20,440 valid values) are compared to the results of external software (for 1000 random points). Considering that the different sample sizes can lead to different accuracies and the inhomogeneity of the area, it is obvious that the results are almost identical.

1. Introduction

Geographic information systems (GISs) have substantially modified the way geospatial information is processed and shared, magnifying the potential to detect and monitor changes in numerous applications of different disciplines in environmental sciences. Hydrology, meteorology, agriculture, forestry, environmental engineering, and geosciences employ GIS applications to obtain complete maps or create spatial distributions for several parameters or variable(s) of interest (VOI) [1,2,3,4,5,6,7,8]. Remote sensing exhibits a pivotal role in numerous interdisciplinary applications, often in combination with ground-based data [9,10,11,12,13]. Remotely sensed products (e.g., surface temperature, precipitation, evapotranspiration, solar radiation, vegetation and water indices, land-use and land-cover (LULC) types, etc.) are retrieved as images (raster format). When it comes to some VOI, images and spatial distributions are powerful tools for conveying information and, subsequently, for facilitating decision-making processes [14,15,16,17,18,19]. The most commonly used statistical metrics in environmental sciences are root mean square error (RMSE), mean bias error (MBE), mean absolute error (MAE), as well as the coefficient of determination and the correlation coefficient [20,21,22,23,24,25]. However, the latter two metrics come with several limitations such as positive bias, sensitivity to the number of independent variables or the sample size, and limited insight into model adequacy. Consequently, several researchers have considered them to be unsuitable for several applications in environmental sciences (i.e., hydrological implementations) [26,27]. The RMSE is the most popular metric. It is also found in the literature as the root mean square difference or deviation (RMSD). The RMSE serves as a short-term evaluator of a model (or method) of interest, by comparing the actual difference between estimated and reference values [28]. However, reporting a single metric is not sufficient to justify the performance of a model, since every metric incorporates uncertainties [29,30,31]. The metrics have some limitations such as the fact that different internal-error components may give similar values of the metric. The fact that RMSE and MAE may produce close values to each other is known as underdetermination of the metrics [23]. Despite the fact that RMSE and MAE are similar metrics, in many cases it is recommended to report both values [26]. The MAE is defined as the mean of the absolute error values, and therefore it is always positive. It is a more robust alternative to the popular RMSE because it is less sensitive to large forecast errors [26,32]. Instead of inflating the larger errors, such as the RMSE does (due to its squaring construct), the MAE increases linearly with an increase in errors [33]. Furthermore, RMSE, MAE, and MBE have the same units as the VOI. This can be straightforward when comparing estimates or the same VOI under specific environmental conditions, but can be misleading to an inexperienced audience where, for example, seasonality affects the magnitude of the error values. That is to say, RMSE, MAE, and MBE metrics depend on the order of magnitude of the VOI [34]. Other ways to address the limitations is to collectively compute the bias metric (MBE) along with the relative or the normalized versions of the metrics presented above [32].

The MBE expresses the bias in the prediction of an examined model. Its usefulness lies in identifying that more steps are needed to correct model bias [35]. Moreover, the MBE is a signed metric; a minus sign depicts that the reference value of the VOI is greater than the model-predicted value and vice versa. Hence, providing the overestimation or underestimation of a model as compared with a reference is an added value of the derivatives of bias metrics (e.g., the normalized formulae), which is not found in the cases of RMSE and MAE metrics [28]. Currently, the common practice of model verification or method validation is to compute a concise set of statistical metrics. Normalized root mean square error (NRMSE), normalized mean absolute error (NMAE), and normalized mean bias error (NMBE) are often used in combination with RMSE, MAE, and MBE [36,37,38,39,40,41,42,43]. Their main contribution is that, contrary to the non-normalized metrics, they are expressed as percentages that enable intercomparisons (e.g., among different seasons) since the metrics are disentangled from the unit scale [28,34].

The significant novelty of this study lies in the fact that the GIS community is still not provided with a reliable and quick suite of automated tools that can produce the mean RMSE, MAE, MBE, NRMSE, NMAE and NMBE of a raster file with measured or predicted values against a reference file (of the VOI), via pairwise cell-by-cell calculations. ArcGIS popular packages (ArcGIS Pro 3, ArcMap 10.8) [44] include only descriptive statistical measures. Despite specific programming constraints, the model builder of ArcGIS Pro 3/ArcMap 10.8 is a useful tool to develop tailored statistical metrics, mostly for feature forms (polygons, lines, and points). This tool is programmed in Python language and provides a graphical user interface (GUI), which facilitates the developing design as well as the improvement of the model. Nevertheless, several constraints appear when it comes to raster modeling. Since automatic computing of statistical errors between raster files is not yet an option, researchers who need those metrics must use one of the following methodological suites: either exporting points from all raster files they wish to compare and using external software (e.g., Excel, SPSS), or importing files in programming platforms such as Matlab, R, etc. and developing the tailored script for their needs [45,46,47,48]. Multiple batches of exporting and importing data and incompatibility issues between ArcGIS Pro 3/ArcMap 10.8 and the external software plus the required expertise in the corresponding software, interfere in their work. In addition, in programming environments such as Matlab, only images of the same size can be compared, therefore, some extra preprocessing is also imperative. Another option would be for users to manually execute a number of different orders from the toolboxes of ArcGIS Pro 3/ArcMap 10.8, along with creating the appropriate algebra expressions in raster calculator. However, this sequence is time-consuming, confusing, and transferability is error-prone similar to most manual suites [49].

There have been several tools developed based on a GIS (mostly in ArcMap and QGIS) in environmental sciences [50,51,52,53,54]. However, only a few researchers have attempted to create metric models for specific implementations by incorporating raster files, for example, Abd-Elrahman et al. [55] who developed raster-based automated models for canopy delineation and canopy size metrics for modeling strawberry dry weight using ArcMap 10.7 model builder [55]. The former, albeit addressing very specialized needs, underlined the necessity of developing automated statistical metrics for raster files in ArcMap. The lack of proper statistical tools in a GIS environment was also highlighted by Jepur et al. [49], who developed three bivariate statistical models (i.e., FR, WoE, and EBF) with raster inputs, using the ArcMap model builder [49]. The models are operated as python scripts and as graphical interface models and can be implemented in geohazard modeling. The latter modeling frame is in the same vein as our envisioning of easily operated, flexible, but still precise and reliable statistical tools in an ArcGIS Pro 3/ArcMap 10.8 environment for a wide range of environmental applications.

Aiming to fill the gap, we developed a toolbox consisting of six automated models that could be inserted and run either in ArcGIS Pro 3 or ArcMap 10.8. The sequence of the needed tools is graphically presented in the Results section. The internal settings and the appropriate map algebra expressions of the models can be displayed in the Edit mode. The corresponding Python scripts can also be easily exported via the Edit mode. The justification of the proposed models is presented through an application on reference evapotranspiration (ETo) for the Peloponnese peninsula in Greece, during different seasons. This application was selected as a challenging combination of a sensitive hydrological VOI (i.e., ETo), examined in a testbed which is considerably inhomogeneous in terms of LULC, altitude, and climate conditions (i.e., the Peloponnese). Moreover, the application was implemented for two different months of 2018 which belong to different seasons.

2. Materials and Methods

2.1. The Developed Statistical Metrics

2.1.1. Equations of the Metrics

The developed models compute the most frequently used statistical measures in interdisciplinary environmental research, namely, RMSE (or RMSD), the signed MBE (or MB), MAE, NRMSE, the signed NMBE (or NMB), and NMAE (

Table 1. The formulae of the developed statistical models.

RMSE	MBE	MAE	NRMSE	NMBE	NMAE
$\sqrt{\frac{1}{\mathrm{n}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{p}}_{\mathrm{i}}-{\mathrm{o}}_{\mathrm{i}}\right)}^2 }$, (1)	$\frac{1}{\mathrm{n}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}\left({\mathrm{p}}_{\mathrm{i}}-{\mathrm{o}}_{\mathrm{i}}\right)$, (2)	$\frac{1}{\mathrm{n}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}\left|{\mathrm{p}}_{\mathrm{i}}-{\mathrm{o}}_{\mathrm{i}}\right|$, (3)	$\frac{\sqrt{\raisebox{1ex}{${{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{p}}_{\mathrm{i}}-{\mathrm{o}}_{\mathrm{i}}\right)}^2$}\!\left/ \!\raisebox{-1ex}{$\mathrm{n}$}\right.}}{\stackrel{-}{\mathrm{o}}}$, (4)	$\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\left({\mathrm{p}}_{\mathrm{i}}-{\mathrm{o}}_{\mathrm{i}}\right)}{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{o}}_{\mathrm{i}}}$, (5)	$\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\left|{\mathrm{p}}_{\mathrm{i}}-{\mathrm{o}}_{\mathrm{i}}\right|}{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{o}}_{\mathrm{i}}}$, (6)

Note: p_i stands for the i-th measured/predicted value, o_i stands for the i-th observed/reference value, $\stackrel{-}{\mathrm{o}}$ is the mean of the observed/reference values, and n stands for the sample size.

, Equations (1)–(6)) [56,57,58].

2.1.2. Model Requirements and Affordances

The mean metric values (for the study area or the area of interest) are computed in the same units as the values of the input raster, except from the normalized metrics (NRMSE, NMBE, and NMAE) which, by default, produce the error as a percentage (value * 100%). Therefore, the only prerequisites of the models are that the under-evaluation (Input) raster file and the reference raster file (Reference) have the same units, cell size, and projection system (

Table 2. Affordances and requirements of the developed statistical models.

Attributes	Required	Not Required	Applicable
Same size of the raster inputs		X
Same no-data pixels of the raster inputs		X
Preprocessing to remove no-data pixels		X
Programming background of users		X
Same cell size of the raster inputs	X
Same projection system of the raster inputs	X
Same units of the raster inputs	X
Inputs in .tiff format	X
Outline of the study area (polygon)	X *
Ability to apply for any study area			X
Ability to insert and run the developed toolbox (ArcGIS Pro 3/ArcMap 10.8)			X
Ability to modify models (free, provided the ARCGIS Pro 3/ArcMap 10.8 packages are already purchased)			X
Ability to operate models as python (v.3 for ArcGIS Pro 3 and v.2.7.18 for ArcMap 10.8) scripts			X

Note: * required only for the normalized metrics.

).

The models are applicable to any type of raster file in .tiff format, as well as to any size of raster, no matter if any no-data pixels are included or their distribution. The different size between the reference and the under-evaluation (Input) raster file is not an issue herein. The models’ computations are executed cell-by-cell. The models select and include in the computations only those pixels which have valid values in both raster files, cutting off any no-data pixels of the input file, as well as the pairwise pixels of the reference file (and vice versa), before the computation. The former attributes make the models widely and interdisciplinary applicable.

2.2. Application on Reference Evapotranspiration Images

2.2.1. Study Area

The area of implementation is the Peloponnese peninsula southwest of central Greece. The terrain is mostly hilly and mountainous with a well-developed hydrographic network, but only a few large rivers [59]. The climate is Mediterranean warm temperate (“Csa”) [60]. The annual climate normals of air temperature, sunshine, and precipitation range between 8 and 20 °C, from 1900 to 3100 h, and from 400 mm to over 2000 mm, respectively [61,62].

2.2.2. Employed Data

The input pairs (Input and Reference) of the models were the same for all models (RMSE, MBE, MAE, NRMSE, NMBE, and NMAE). Input included raster files of ETo computed by several empirical methods and MODIS ET files (

Table 3. Input ETo (mm) files in raster format (.tiff) for the December and August applications of the developed statistical models.

December 2018
Inputs	Hargreaves Samani	Hansen	Makkink	MODIS ET	FAO PM
August 2018
Inputs	De Bruin	Blaney Criddle	Hammon 2	MODIS ET	FAO PM

Note: The names of the empirical methods were used as in [63].

) [63,64,65,66].

2.2.3. The Pairwise Comparison

The FAO PM was used as the references [67,68,69,70]. The MODIS ET raster files, as well as the raster files of the empirical methods (in tiff format), were retrieved ready-to-use from two recent works on ETo across the Peloponnese peninsula [71,72]. Models were recreated or refined thereafter and reached their optimal form, as presented in this study (Section 3.1). Each model used two raster files of the VOI as input: the under-evaluation file (Input) and the reference file (reference), with different sizes and variability in terms of number and position of the no-data pixels. Some selected pairs were used to check the model prediction of the smallest and the largest ranges of errors, because the errors between the particular pairs of methods were the lowest and largest reported, respectively [71,72].

Moreover, the outline of the study area (polygon file) was also required in the normalized metrics to mask the produced images. In addition, several raster files that were internally produced were also used to mask the output image of a successive step. The cell size of all the raster files in the application was 1 km. The models computed the mean metric (error) values of raster files with at least 20,440 valid data pixels each, in some seconds. They could also be run through the edit mode, for real-time visual observation of the executing steps in both ArcGIS Pro 3 and ArcMap 10.8. The reliability of the models was verified via two repeated runs of each model with the same parameters. The outputs were identical.

2.2.4. Retrieval of the Output Metric Values

As soon as each model’s execution was completed, the produced metric value was available to the user in one out of two representations. The mean error value of the non-normalized metrics (RMSE, MBE, and MAE) is the “Mean” value displayed in the Layer Properties > Source > Statistics > Mean of the output raster file (in the Contents/Table of Contents pane). The added value of the models is that not only the mean, but also the standard deviation (sd), and the minimum and maximum values of the corresponding metric are computed and displayed in the Statistics section. As far as the normalized metrics are concerned, the models produce an output raster file with only one value: the output value of the metric, which is displayed in the Table of Contents/Contents pane. In cases of the bias metrics (MBE and NMBE), as already explained, the output values can be positive or negative. Those representations are presented in Section 3.1.2.

2.2.5. Metric Values Using External Software

After acquiring the metric values for all the examined pairs of ETo (via executing the developed models), we created 1000 random points with the limitation of distance greater than 1500 m (pixel’s hypotenuse) among the points (Figure 1). This setting ensured that sampling the same pixel twice would be prevented; moreover, it helped to better distribute the sampling points across the study area. The next step was to extract ETo (or MODIS ET) values to points from each one of the raster inputs (see Table 3), and then to delete the points for which at least one invalid value occurred. At last, 945 points with valid ETo (or MODIS ET) values constituted the second sample. Although almost the entire sample space (at least 20,440 valid pixels) participated in the computation for the models, a sample size of 945 was large enough to be considered representative. However, the difference in the order of magnitude between the two samples might lead to discrepancies between the corresponding metric values. Although this is a limitation, it is rational that the models provide more accurate results since they employ almost the entire population (e.g., ETo values) in the calculation.

The calculators of the “Agricultural and Meteorological Software (AgriMetSoft)” (Seoul, South Korea) were utilized for computing the RMSE, MAE, MBE, and NRMSE for the 945 points [73]. The NMBE and NMAE metrics were constructed in Microsoft Excel 365 (Washington, DC, USA), since they were not available [45]. The aforementioned software packages were selected as they are reliable and well-established. The former software prerequired using an accounting spreadsheet format (e.g., in Excel) to organize the values in pairs and to set the desirable number of decimals, then to paste the under-comparison pairs in a form and to execute the metric of interest. Then, the output value was automatically produced and rounded to two or three decimal places without user intervention.

In the case of Microsoft Excel, the decimal place option is not a constraint, but the computation procedure needs to be manually executed for each pair, a process encompassing intensive workload. The output values produced by the external software (i.e., Agricultural and Meteorological Software, and Microsoft Excel) are denoted by “soft” index (e.g., RMSE soft), whereas the models’ values are denoted by “mod” index (e.g., RMSE mod) in the Results section.

3. Results

3.1. The Developed Statistical Models

3.1.1. Graphical Representation

The developed statistical models are graphically presented (Figure 2, Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7). The employed tools, the map algebra expressions, and the settings of each step are included. The list is as following: root mean square error (RMSE) (Figure 2), mean bias error (MBE signed) (Figure 3), mean absolute error (MAE) (Figure 4), normalized root mean square error (NRMSE) (Figure 5), normalized mean bias error (NMBE signed) (Figure 6), and normalized mean absolute error (NMAE) (Figure 7).

3.1.2. Explanation of Variables and Execution

To run the models, users must previously insert the Statistical Errors Raster Toolbox in either ArcGIS Pro 3 or ArcMap 10.8. Then, in the former case, they select the Tag View > Geoprocessing, type the model they wish to run in the search field (e.g., “MAE”), and select it from the searching results. Alternatively, they can insert the toolbox from the Catalog pane (select the tag insert, right-click the option “toolboxes” in the Contents pane, select “Add Toolbox”, then upload the file “statistical errors raster toolbox.tbx”, (see Supplementary Materials). In the case of ArcMap, the user can select the tag Geoprocessing > ArcToolbox, right-click and select “Add Toolbox”, then upload the file “statistical errors raster toolbox.tbx”. The models can be also found via the Catalog pane. Users need only to double-click the model they wish to run after unfolding the toolbox, or right-click it and select Open.

The non-normalized models (RMSE, MAE, and MBE) have similar user interfaces consisted of three fields which are displayed on screen as soon as the the user clicks Open (presented in Figure 8a for the MAE model). The first two fields are inputs (i.e., “Input” is the under-evaluation raster file and “Reference” is the reference file of the VOI), while in the third one, the user can name the output raster file (e.g., “MAE”). The running-time for MAE in ArcGIS Pro 3 is 2 s (see Figure S1 in Supplementary Materials), while for MBE and RMSE, the running times are 3 s and 8 s, respectively. The corresponding running time for the latter is shorter in ArcMap 10.8. As soon as running is completed, the output raster is by default displayed in the (ArcGIS Pro 3/ArcMap 10.8) viewer. One needs only to right-click the “MAE” layer in the Contents/Table of Contents pane, select Layer Properties > Source > Statistics, and retrieve the displayed mean value (e.g., 0.499621 mm, Figure 8b).

The normalized models (NRMSE, NMAE, and NMBE) have similar user interfaces, with four fields (Figure 8c). In addition to the outline of the study area/area of interest (polygon file) which is new, the remaining fields are in the same vein as already described (Input, Reference, and Output). The running time for either NMAE or NMBE in ArcGIS Pro 3 is 12 s, while it is 41 s for NRMSE (see Figure S2 in Supplementary Materials). In ArcMap 10.8, the times are much shorter. Once the execution is completed, the output raster (e.g., “NMAE”) is by default displayed in the (ArcGIS Pro 3/ArcMap 10.8) viewer. One needs only to read/copy the value of the NMAE (single-value) layer in the Contents/Table of Contents pane (e.g., 0.325081 or 32.5%, Figure 8d).

3.2. The ETo Implementation

Metric Values Computed bUsing the Models and Using External Software

The statistical errors of ETo computed using the developed models (for images with at least 20,440 valid pixels), and using the external software (945 random points with valid values) are presented in

Table 4. Statistical values of daily mean ETo for December 2018 using the models and using external software.

December 2018
	Hargreaves Samani—FAO PM	Hansen—FAO PM	Makkink—FAO PM	MODIS ET—FAO PM	Hansen—Hargreaves Samani
RMSE mod	0.5031	0.4406	0.5590	0.5304	0.0893
RMSE soft	0.5080	0.4440	0.5620	0.5320	0.0910
MBE mod	−0.4297	−0.3499	−0.4906	−0.4656	0.0794
MBE soft	−0.4340	−0.3520	−0.4920	−0.4640	0.0820
MAE mod	0.4448	0.3696	0.4996	0.4728	0.0795
MAE soft	0.4470	0.3700	0.5000	0.4730	0.0820
NRMSE mod	0.3724	0.3261	0.4139	0.3930	0.0960
NRMSE soft	0.3770	0.3290	0.4170	0.3940	0.0990
NMBE mod	−0.3180	−0.2590	−0.3630	−0.3450	0.0862
NMBE soft	−0.3220	−0.2610	−0.3650	−0.3440	0.0895
NMAE mod	0.3292	0.2735	0.3698	0.3505	0.0862
NMAE soft	0.3313	0.2744	0.3708	0.3504	0.0895

Note: “mod” stands for model (metrics calculated cell-by-cell for at least 20,440 cells with valid values), “soft” stands for external software (metrics calculated point-by-point for 945 random points with valid values). RMSE, MBE, and MAE are given in mm, whereas the other metrics are unitless (* 100%).

and

Table 5. Statistical values of daily mean ETo for August 2018 using the models and using external software.

August 2018
	De Bruin—FAO PM	Blaney Criddle—FAO PM	Hammon 2—FAO PM	MODIS ET—FAO PM	Hansen—Hargreaves Samani
RMSE mod	0.6410	1.0660	1.6760	3.2420	0.5292
RMSE soft	0.6530	1.0420	1.6370	3.2320	0.5210
MBE mod	−0.5869	0.9570	1.1967	−3.1320	−0.4870
MBE soft	−0.5990	0.9310	1.1460	−3.1330	−0.4780
MAE mod	0.5869	0.9710	1.3850	3.1332	0.4889
MAE soft	0.5990	0.9440	1.3460	3.1330	0.4790
NRMSE mod	0.1231	0.2048	0.3221	0.6229	0.1148
NRMSE soft	0.1250	0.2000	0.3140	0.6200	0.1130
NMBE mod	−0.1127	0.1839	0.2298	−0.6017	−0.1058
NMBE soft	−0.1150	0.1790	0.2200	−0.6010	−0.1043
NMAE mod	0.1127	0.1865	0.2661	0.6019	0.1061
NMAE soft	0.1149	0.1811	0.2583	0.6011	0.1045

Note: “mod” stands for model (metrics calculated cell-by-cell for at least 20,440 cells with valid values), “soft” stands for external software (metrics calculated point-by-point for 945 random points with valid values). RMSE, MBE, and MAE are given in mm, whereas the other metrics are unitless (* 100%).

, for December 2018 and August 2018, respectively.

The values of the metrics (RMSE, MAE, MBE, NRMSE, NMAE, and NMBE) from Table 4 and Table 5 are graphically presented via the histogram in Figure 9. There is a pair of bars (e.g., RMSE via model and RMSE via external software) for each statistical metric, as annotated in Figure 9.

4. Discussion

The Statistical Errors Raster Toolbox consists of six statistical models of error metrics (i.e., RMSE, MBE, MAE, NRMSE, NMBE, and NMAE) which, as a rule, are used in environmental sciences for a wide range of applications. The graphical representations (model builder environment, see Figure 2, Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7) of the models, along with the corresponding toolbox (see Statistical Errors Raster Toolbox in Supplementary Materials) are available. The added value of the models is that they enable computations directly between raster files (tiff. format). This means that it is not necessary for users to utilize external software for the computations. External software usually comes with side constraints such as data incompatibility, successive batches of transferring data, prerequired expertise in the operation of each package, programming skills, and devoting of time and labor. Moreover, the models employ almost the total sample space in the computations, whereas, in most statistical or accounting packages (e.g., SPSS, Excel), point values are used, aggravating the accuracy of the output metric value. Therefore, external software may be a serious barrier for ArcGIS Pro 3/ ArcMap 10.8 users in carrying out their computation of interest. The developed statistical metrics are, instead, easy-to-run even by basic users, and easy to modify in either model builder (see Figure 2, Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7) or python environment. Furthermore, they are automated. This means that after the Statistical Errors Raster Toolbox is inserted, the models are ready to run. Each output is available in some seconds (ranging from 2 to 12 s in ArcGIS Pro 3, with the exception of NRMSE which needs 41 s due to its increased complexity), (see Figures S1 and S2 in Supplementary Materials). Their accuracy is the best possible since they use the total of the valid pixels for the computation and the significant digits are more than sufficient (e.g., see Figure 8b,d). The output of each model is a raster file. The metric value can be easily retrieved: it is either displayed in the Contents/Table of Contents pane as the single value of the output raster (for NRMSE, NMBE, and NMAE) or it is displayed as the “Mean” in the Statistics section (Layer Properties > Source) of the output raster file (for RMSE, MBE, and MAE). In the case of the latter, not only the mean metric value, but also the minimum and maximum metric values plus the standard deviation are computed and displayed in the Statistics section. This information is useful insofar large raster files are employed. The developed models are not yet an option in ArcGIS Pro 3/ArcMap 10.8 software, or in any other GIS environment, although they are pivotal metrics in environmental sciences and engineering research.

The models were applied in a two-fold implementation for daily mean ETo (VOI) across the Peloponnese peninsula, Greece, for the months of August and December 2018. The models employed at least 20,440 valid pixels (almost the total sample space) to compute each mean metric value of the area of interest. In place of the external software, a trustworthy free online metric calculator package (AgriMetSoft) and the well-established Microsoft Excel package were used. They employed 945 valid pixel values out of the 1000 points which had been randomly generated for the Peloponnese. The application exploited factors which affect ETo such as the different seasonality (August/December), the variability of LULC, and altitude and environmental conditions across the area of application. There were some limitations such as the incapability to statistically compare the outputs due to their inadequate number (small sample size), the automatic rounding of the values by the external software, and the difference in the order of magnitude between the two sample sizes (i.e., 20,440 vs. 945). However, the double representation of the outputs: in numbers (see Table 4 and Table 5) and via a histogram of paired bars for each metric and case (Figure 9), make the similarity of the results obvious, at least for two decimal places (three significant digits) in most cases, for a VOI (ETo) measured in mm. Based on the points already discussed, it is rational that the accuracy of the developed model outputs is greater than that of the employed external software. It would also be useful for the models to be applied for a suite of different parameters (i.e., VOI, study area, resolution, raster size, and no-data pixel distribution), for a demonstration of their flexibility and competence. An alternative comparison could be performed against other external software such as Matlab, R, SPSS, etc. The six developed statistical metrics are reliable and accurate automated tools, and easy-to-run by the broadest part of GIS users in different disciplines. The former underlines that the developed Statistical Errors Raster Toolbox which runs in both ArcGIS Pro 3 and ArcMap 10.8 would ease users’ workloads. Moreover, the Statistical Errors Raster Toolbox could constitute an integrated solution for raster file statistical analyses with limited requirements.

5. Conclusions

The lack of proper statistical models to compute error metrics for environmental sciences in any GIS environment, prompted the development of the Statistical Errors Raster Toolbox, which includes six models of the most popular statistical metrics (RMSE, MBE, MAE, NRMSE, NMBE, and NMAE). The added value of the models is that they enable computations directly between raster files (tiff. format), producing an output raster file. Their accuracy is the best possible, since they use the total of the pairwise-valid pixels (almost the entire sample space) for the computations and the significant digits of the output value are more than sufficient (see the precision and scale for float or double variables in ArcGIS). An application implemented for ETo raster files of at least 20,440 valid pixels for the Peloponnese peninsula, Greece, also employing remote sensing (MODIS) input data, showed that the values produced by the models are almost identical to the values obtained by using external software. The discrepancies are probably due to the lower accuracy (smaller sample size, rounding to two or three decimal places) of the external software. The models can be implemented for any area of interest and any size of raster inputs, no matter if the compared raster files have different sizes or different no-data pixels. The models have very limited requirements to run effectively, i.e., same cell size and projection system of the raster inputs, same units of the VOI, plus a polygon outline of the area of interest (the latter in the cases of the normalized metrics). They run only in some seconds, even by inexperienced users, through either the open or the edit mode (real-time observation of running parts). The toolbox can be inserted and run either in ArcGIS Pro 3 or ArcMap 10.8. The potential of also running the models in relevant packages (e.g., ERDAS IMAGINE) would be a future perspective. The Statistical Errors Raster Toolbox consisting of six automated models could facilitate scientists in interdisciplinary statistical raster analysis.