Towards Smart Big Weather Data Management ^†

Abstract

Smart management of weather data is pivotal to achieving sustainable agriculture since weather monitoring is linked to crop water requirement estimation and consequently to efficient irrigation systems. Advances in technologies such as remote sensing and the Internet of Things (IoT) have led to the generation of this data with a high temporal resolution which requires adequate infrastructure and processing tools to gain insights from it. To this end, this paper presents a smart weather data management system composed of three layers: the data acquisition layer, the data storage layer, and the application layer. The data can be sourced from station sensors, real-time IoT sensors, third-party services (APIs), or manually imported from files. It is then checked for errors and missing values before being stored using the distributed database MongoDB. The platform provides various services related to weather data: (i) forecast univariate weather time series, (ii) perform advanced analysis and visualization, (iii) use machine learning to estimate and model important climatic parameters such as the reference evapotranspiration (ET${}_0$) estimation using the XGBoost model (R${}^2$ = 0.96 and RMSE = 0.39). As part of a test phase, the system uses data from a meteorological station installed in the study area in Morocco.

1. Introduction

Since the industrial revolution, our planet is facing real challenges related to climate change [1]. Effects of this latter are going to get worse as we go into the future, especially in the context of a high rate of population growth [2] which reveals another type of issue related to food security. Agriculture is the concerned sector, and agricultural management practices must be optimized to meet the increasing food demand. One step toward this is optimizing water resources usage, which is used mainly in irrigation. There are several types of irrigation, such as surface irrigation, which is the most prevalent mode of irrigation worldwide [3] and pressurized irrigation that requires energy in order to supply water. Whatever the type of irrigation, efficiency can’t be achieved without knowing the right amount of water to apply and when to apply it [4]. To achieve this, estimation of the evapotranspiration is pivotal. It is the sum of the evaporation from the soil surface and the transpiration from plants. The evapotranspiration of a crop (ET${}_c$) is estimated by multiplying the crop coefficient (K${}_c$) and the reference evapotranspiration (ET${}_0$) which represents the rate of evapotranspiration for a reference crop (grass) with known properties obtained by monitoring some meteorological parameters such as air temperature, net solar radiation, humidity, and wind speed. Therefore, weather monitoring is essential for accurately managing irrigation water and, in turn, achieve sustainable agriculture.

Thanks to advances in science and technology, we are currently able to collect weather data at high spatial and temporal resolution and with low-cost [5,6,7,8,9,10,11]. This huge amount of generated data is unprecedented, and it brings the notion of big data to this field, which requires adequate infrastructure and processing methods.

This paper presents a web platform to store and process this abundant resource (data) using big data analytics and artificial intelligence algorithms.

2. Materials and Methods

2.1. Study Area and Data

The experimental site (Figure 1) is located 40 km east of Marrakesh city in Morocco (31°39${}^{\prime }$33.68${}^{\prime }$${}^{\prime }$ N, 7°36${}^{\prime }$23.586${}^{\prime }$${}^{\prime }$ W, 582 m above mean sea level using the World Geodetic Coordinate System (WGS84 [12])). It is an irrigated area of about 2800 ha, which is almost flat. It has a Mediterranean semiarid climate, with around 250 mm of average annual rainfall [13,14] generally recorded between November and the end of April and average annual evapotranspiration (ET${}_0$) of 1600 mm [15].

The collected data from the weather station installed in the study area covers the period between January 3, 2013, to December 31, 2020 (

Table 1. Meteorological station data description.

Variables	Description	Unit
R3_Dv	Wind direction	Degree
R3_Hr	Relative humidity	No unit
R3_Rg	Global solar radiation	W m${}^{-2}$
R3_Tair	Air temperature	°C
R3_Vv	Wind speed	m s${}^{-1}$
R3_P30m	Rainfall	mm

) at half-hour scale. A detailed description of the different sensors used in the station can be found in previous work conducted in the same study area [17,18].

2.2. Proposed Smart Weather Data Management Platform

2.2.1. Overview

The proposed platform adopts a service-oriented architecture providing services that cover all four types of data analytics from descriptive data analysis to prescriptive data analysis (Figure 2).

It consists of three layers: the data acquisition layer, the data storage layer, and the application layer (Figure 3). The data can be collected from heterogeneous sources such as meteorological station sensors, real-time IoT-based weather stations, reanalysis data, third-party meteorological services or Application Programming Interfaces (APIs), or manually imported from physical files (CSV, Excel, etc.). This data is then checked and preprocessed to handle missing values, before being stored using the MongoDB NoSQL database. This database represents a suitable solution for our use case, given its ability to handle real-time data, querying, and retrieving large volumes of data, in addition to its scalability and schema-less characteristics. In contrast, fault tolerance issues are a weakness of MongoDB and are the case for almost all distributed databases as well.

2.2.2. Forecasting Service

The forecasting service helps predict the future evolution of weather (air temperature, solar radiation, relative humidity, etc.). It initially uses the Facebook Prophet model [19] given the performance it shows when compared with a Long Short-Term Memory (LSTM [20]) neural network architecture when applied to the same meteorological station data we used [21].

The platform enables the user to specify the number of years needed to train and test the model performance and the time period for the forecast.

Results for univariate time series forecasting were evaluated on the data available in the platform database described in Table 1. The evaluation metrics used were: the Coefficient of determination Equation (1) and Root Mean Squared Error Equation (2). Results are presented in Section 3.1.

$$R^2=1-\frac{{\sum }_1^n{(y_i-{\widehat{y}}_i)}^2}{{\sum }_1^n{(y_i-{\overline{y}}_i)}^2}$$

(1)

$$RMSE=\sqrt{\frac{1}{n}\sum _1^n{(y_i-{\widehat{y}}_i)}^2}$$

(2)

2.2.3. Weather Data Analysis and Visualization Service

This service provides multiple visualization options such as line charts for the evolution of weather time series and also provides data analysis of stored data. One example of such analysis is the generation of the Pearson correlation coefficient matrix calculated using Equation (3) which is then used to investigate the relationship between weather variables.

$$r=\frac{\sum \left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)}{\sqrt{\sum {\left(x_i-\overline{x}\right)}^2\sum {\left(y_i-\overline{y}\right)}^2}}$$

(3)

In the proposed platform, the user choose the target variables and the correlation matrix will then be generated automatically (Figure 4).

Additionally, the platform uses machine learning for modeling and estimation of important parameters used in agriculture such as the evapotranspiration. The flowchart for Figure 5 presents an approach that uses the FAO Penman-Monteith ET${}_0$ [22] as a reference method to learn to estimate the reference evapotranspiration from raw station metrological data. Results of this approach are presented in Section 3.2.

3. Results and Discussion

3.1. Univariate Time Series Forecasting Service

Table 2. Performance of univariate forecasting service.

Metric / Parameter	T${}_{\mathit{a}}$	R${}_{\mathit{g}}$	H${}_{\mathit{r}}$	Wind Speed
R${}^2$	0.82	0.83	0.48	0.18
RMSE	3.61	114.23	16.73	0.94

presents results for a one-year forecast of the meteorological time series (2020) given the historical data (from 2013 to 2019) at an hourly scale. The concerned parameters were: Air temperature (T${}_a$), Global solar radiation R${}_g$, Relative humidity H${}_r$, and Wind speed.

3.2. Estimation of Climatic Parameters Using Machine Learning

To estimate the ET${}_0$, the performance of the XGBoost [23] machine learning model was evaluated on the data available on the platform, which is used to generate the training (80%) and testing (20%) split sets. This model gives a high coefficient of determination R${}^2$ = 0.96. This means that 96% ET${}_0$ is explained by the trained model. In turn, the RMSE was 0.39. Also, Figure 6 shows a positive linear scatter which indicates a good fit for the model. Code implementation was done using the DST library [24].

Despite the good results obtained, more studies can be conducted to investigate different machine learning models for both forecasting and regression tasks and integrate them into the platform. In the same sense, current proposed services can be optimized and enriched depending on the challenges and problems faced in the deployment phase. In data storage, for example, other technologies can be investigated when it comes to scaling up the system or in dealing with batch processing tasks. A solution to investigate could be the Hadoop ecosystem (Spark, MapReduce, Hive, etc.).

4. Conclusions

In this paper, we proposed a platform based on artificial intelligence and big data analytics as a way to manage weather data efficiently to assist decision-making in agriculture. The platform offers various services for both farmers and policymakers such as weather data visualization, analysis and estimation. It was tested using data from the meteorological station installed in our study areas, covering the period between 2013 and 2020. The platform will be enriched in future work with other services related to implementing smart agricultural practices.