Electric Consumption Forecast for Ships Using Multivariate Bayesian Optimization-SE-CNN-LSTM

Abstract

Many studies on reducing greenhouse gas emissions from ships have been conducted to reduce environmental pollution. Reducing the fuel oil consumption of traditional and green ships is a key focus of these studies. The fuel oil consumption of the ship depends on electric loads. Thus, ship power load estimation is necessary to develop methods for reducing the fuel oil consumption of ships. However, data accessibility for ship power load estimation is low, limiting the number of relevant studies. This study proposes a model for estimating the actual power load of ships using squeeze and excitation (SE), a convolutional neural network (CNN), and long short-term memory (LSTM). The electric load, power generated by the generator, power consumption of the reefer container, rudder angle, water speed, wind speed, and wind angle of a ship were measured in 10-minute increments for approximately 145 d. The existing parallel and direct CNN-LSTM power load estimation models were used to evaluate the performance of the proposed model. The proposed model had the lowest root mean square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE), demonstrating the best ship power load estimation performance compared to existing power load estimation models.

1. Introduction

Various methods for reducing greenhouse gas emissions from ships have been studied in recent years. In particular, research on reducing fuel oil consumption of traditional ships and green ships, which do not use internal combustion engines but consume fuel oil, is increasingly being conducted [1,2,3,4,5]. Forecasts representing the actual load data of the ships are essential for these studies because they can evaluate the amount of electrical energy that the algorithm can reduce by estimating the power load of ships.

Green ships use fuel cells, batteries, eco-friendly power sources, and supercapacitors as power sources [6,7,8]. Power loads in traditional ships are classified into general power loads (consistently used) and intermittent heavy loads. Owing to the high electricity consumption of heavy loads, ensuring a sufficient fuel cell and battery capacity is necessary for the smooth operation of green ships [8,9,10,11]. Therefore, estimating the power load of ships is necessary to promote the utilization of green ships.

Unlike those related to the power load in ships [12,13], the trend, seasonality, and periodicity of shore power demand can be determined conveniently. Moreover, many studies have been conducted on regression, fuzzy estimation, random forest, support vector machines, and power demand estimation using artificial neural networks owing to the high accessibility of shore power demand data [14,15]. The power load in ships is intermittently high owing to heavy loads, such as the bow thrusters and cargo pumps [16]. Furthermore, research on ship power load estimation is limited because of low data accessibility.

Therefore, this study proposes a model for estimating the actual power load of ships using squeeze and excitation (SE), a convolutional neural network (CNN), and long short-term memory (LSTM). Moreover, this study used the parallel and direct CNN-LSTM power load estimation models presented in previous studies to evaluate the performance of the proposed model.

2. Theoretical Background

2.1. Convolutional Neural Network (CNN)

A CNN can analyze data features or patterns and can be classified as one-, two-, or three-dimensional (1D, 2D, or 3D), making them an excellent tool for analyzing time series data with a 1D structure, images, and video data, respectively. The ship data used in this study are time series data with a 1D structure; therefore, a 1D CNN is used to examine the spatial features within the ship data. A CNN comprises convolution and pooling layers [17,18]. Filters are used in the convolution layer to extract features of the input data. The following formula expresses the operation occurring in the convolution layer:

$$x_k^l=f(b_k^l+{\sum }_{i=1}^{N_{l-1}}{w}_{ik}^{l-1}{y}_i^{l-1}),$$

(1)

where $w_{ik}^{l-1}$ is the weight value of the vectors, $y_i^{l-1}$ is the vector value input into the convolution layer operation, $b_k^l$ represents the bias value, $x_k^l$ is the value calculated using the activation function, and $f$ is an activation function.

2.2. Long Short-Term Memory (LSTM)

A recurrent neural network (RNN) is a framework for processing time series data. It analyzes the relationship between the preceding and present data and estimates the succeeding data. However, a long-term dependency problem occurs, and past information is lost. LSTM, comprising a cell state, forget gate, input gate, and output gate, uses cell states to improve the limitations of RNNs. The cell state ensures that the input information remains unchanged. Figure 1 shows the inner structure of LSTM.

The forget gate calculates the output of the previous cell and current input data and removes them when judged to be unnecessary values. The input gate determines the information to be stored in the cell. The output gate updates the state values at a specific point in time. LSTM utilizes the following equations to perform operations:

$$f_t=\sigma \left(W_f\cdot \left[h_{t-1}, x_t\right]+b_f\right)$$

(2)

$$i_t=\sigma \left(W_t\cdot \left[h_{t-1}, x_t\right]+b_i\right)$$

(3)

$$\widehat{C_t}=tanh\left(W_c\cdot \left[h_{t-1}, x_t\right]+b_c\right)$$

(4)

$$C_t=f_t\ast \widehat{C_t}+i_t\ast \widehat{C_t}$$

(5)

$$O_t=\sigma \left(W_o\cdot \left[h_{t-1}, x_t\right]+b_o\right)$$

(6)

$$h_t=O_t\ast tanh\left(C_t\right),$$

(7)

where $f_t$ is the forget gate, $i_t$ is the input gate, $O_t$ is the output gate, $C_t$ is the cell state or memory cell, $h_{t-1}$ is the previous state, $\widehat{C_t}$ is the candidate for the cell state at timestamp (t), and $h_t$ is the output state or next hidden state. $\sigma$ and $tanh$ are the activation functions sigmoid and hyperbolic tangent, respectively. $W_t$, $W_f$, $W_c$, $W_o$ are the respective weights, and $b_f$, $b_i$, $b_c$, $b_o$ are the biases. $h_{t-1}$ is the output of the previous LSTM block, and $x_t$ is the input at the current timestep.

2.3. Squeeze and Excitation (SE) Block

The SE block, which is behind the CNN [19], comprises a squeeze that extracts only relevant information and an excitation that recalibrates the squeezed data. Using the SE block enables features output by the CNN to be analyzed to emphasize significant features and suppress the insignificant ones. Global average pooling can be performed for squeeze and expressed with the following equation:

$$z_c=F_{sq}\left(u_c\right)=\frac{1}{W}{\sum }_{k=j}^W{u}_c\left(i,j\right),$$

(8)

where $z_c$ is the data resulting from the squeeze operations with the data input into the $c$ channel, $u_c$ is the $c$ channel data of the input data, $F_{sq}$ is the squeeze operation, and $W$ is the size of the feature map. The squeezed data proceed to excitation, expressed as follows:

$$s=F_{ex}\left(z\right)=f_{sig}\left(W_2{f}_{relu}\left(W_1{z}_c\right)\right),$$

(9)

where $W_1 is\in {ℝ}^{\frac{C}{r}\times C}$, $W_2 is\in {ℝ}^{C\times \frac{C}{r}}$, $W_1$ is a layer that reduces dimensions according to the $r$ ratio, and $W_2$ is a layer that re-increases the data dimensions according to the $r$ ratio. Further, $f_{relu}$ refers to the rectified linear unit (ReLU) activation function, and $f_{sig}$ is the sigmoid function.

2.4. Bayesian Optimization

The deep learning model consists of various hyperparameters, such as the number of neurons, learning rate, and batch size. Selecting the value of the hyperparameters is necessary to enhance the performance of the deep learning model. Grid or random search is typically used to find the appropriate hyperparameter values for the deep learning model. Grid search algorithms are characterized by increasing the search space with increasing the number of hyperparameters [20], indicating a disadvantage of requiring a substantial amount of central processing unit (CPU) or graphics processing unit (GPU) resources. Furthermore, random search algorithm functions are better than grid search algorithms to enhance the probability to find optimized hyperparameters but have lower performance than Bayesian optimization algorithms [21]. The field of machine learning uses intelligence optimization algorithms, such as the genetic algorithm (GA), simulated annealing algorithm (SA), particle swarm optimization (PSO), and Bayesian optimization algorithm. The Bayesian optimization algorithm has the advantage of finding optimized points while utilizing minimal iteration and computer resources [22,23,24,25,26,27,28]. In addition, it can determine the optimal solution to an unknown object function by creating an alternative model for the object function and hyperparameter pair [29,30,31]. It updates the hyperparameters sequentially and searches for the optimal hyperparameter combination.

$$p^{*}=\underset{p\in P}{\mathrm{argmin}}loss\left(p\right),$$

(10)

where $P$ represents the set of all hyperparameters, $p$ refers to the hyperparameter combinations that can be made using $P$, $p^{*}$ is the hyperparameter combinations derived through optimization, and $loss(·)$ is the object function for the Bayesian optimization. If the commonly used root mean square error (RMSE) is used as the loss function, the following equation can be used:

$$loss\left(p_j\right)=\sqrt{\frac{{\sum }_{i=1}^n{\left(\widehat{y_i}\left(p_j\right)-y_i\right)}^2}{n}},$$

(11)

where $p_j$ refers to the jth combination among all hyperparameter combinations, $y_i$ refers to the ith true value, and $\widehat{y_i}$ is the ith value estimated by the deep learning model using $p_j$. An observed dataset, $D$, is produced for the Bayesian optimization operations, expressed as follows:

$$D=\left[\left(x_{i, } y_i\right), \left(x_{i+1, } y_{i+1}\right)\cdots \right],$$

(12)

where $x_{i }$ is the $i$th hyperparameter combination and $y_i$ is the error value occurring with the deep learning model using the $i$th hyperparameter combination. The dataset, $D$, can be used to obtain the posterior probability with the following equations:

$$p\left(loss\right)=G\left(loss;\mu , K\right),$$

(13)

$$P\left(\left.loss\right|D\right)=G\left(loss;{\mu }_{\left.loss\right|D}, K_{\left.loss\right|D}\right),$$

(14)

where G indicates a Gaussian distribution, $\mu$ refers to the mean, and $K$ represents the variance. Bayesian optimization uses previous sampling points to predict the form of the object function and finds the point where the hyperparameter values can derive the maximum result desired by the user. Figure 2 shows the flowchart of the Bayesian optimization algorithm.

2.5. Data Description

The power load of a container ship comprises the power load of the equipment used to propel the ship and hotel load. Determining the factors affecting the power load of the ship is essential for load estimation. Ships must maintain stable sailing routes; thus, all ships continuously control their rudder angle. Some ships use an electric hydraulic system (EHS) for rudder angle control. The EHS operates the hydraulic pump using an electric motor, which controls the rudder angle by moving the hydraulic cylinder. It utilizes the multivariate auto-regressive eXogeneous (MARX) model for rudder control, expressed as follows:

$$X\left(n\right)={\sum }_{m=1}^{M}A\left(m\right)X\left(n-m\right)+{\sum }_{m=1}^{M}B\left(m\right)Y\left(n-m\right)+U\left(n\right),$$

(15)

where $X\left(n\right)$ is a 2D vector that includes yaw and roll. $U\left(n\right)$ is the Gaussian white noise, and $X\left(n-m\right)$ and $Y\left(n-m\right)$ are the differences between the measured input and output vector values. $M$ is the $M$th value of command. Furthermore, the optimal control rule of the MARX model can be expressed as follows:

$$Y\left(n\right)=GZ\left(n\right),$$

(16)

where $Y\left(n\right)$ is the optimal control law, $G$ is the stationary gain, and $Z\left(n\right)$ is the state vector.

The rudder angle set using the MARX model is maintained continuously. However, ships are subject to constant wave and wind effects, and the rudder angle is subject to change during severe wave and wind conditions. We collected data from actual merchant ships. The data measured in this study for ship load estimation are electric load, power generated by the generator, power consumption of the reefer container, rudder angle, water speed, wind speed, and wind angle. Every ship exploits the rudder angle for maneuvering, and the rudder control system requires electricity to maintain its system. According to the value of ship resistance, the rudder control system’s electricity consumption varies [32]. Additionally, water speed, wind speed, wind angle, and rudder angle are classified as resistances on the vessel [33]; therefore, we selected these data for this study because they can directly affect a ship’s electricity load. The data were measured and acquired in 10-min increments for approximately 145 d. A total of 20,935 data entries were used in this study.

Table 1. List and features of the selected data.

Attribute	Unit	Range	Average
Electric Load	kW	585–15,200	1567.38
No. 1 Diesel Generator Power	kW	0–3800	240.98
No. 2 Diesel Generator Power	kW	0–3800	604.42
No. 3 Diesel Generator Power	kW	0–3800	323.9
No. 4 Diesel Generator Power	kW	0–3800	401.07
No. 1 Reefer Container Load	kW	0–130	14.19
No. 2 Reefer Container Load	kW	0–130	12.37
No. 3 Reefer Container Load	kW	0–130	24.17
No. 4 Reefer Container Load	kW	0–130	22.21
No. 5 Reefer Container Load	kW	0–130	21.51
No. 6 Reefer Container Load	kW	0–130	19.76
Rudder Angle	Degree	−38–38°	−1.31
Water Speed	m·s−1	−4.36–16.66	9.85
Wind Speed	m·s−1	0–24.8	6.23
Wind Angle	Degree	0–360°	155.08

lists the data used in this study.

Figure 3 shows the power load data of the ship measured in 10-min increments, with heavy loading occurring intermittently.

3. Proposed Model

3.1. Electric Consumption Forecast for Ships by a Multivariate CNN-LSTM Network with SE Using Bayesian Optimization

The datasets were scaled to accelerate model training. We selected the MinMaxScaler, expressed as follows:

$$MinMaxScaler:x^{\prime }=\frac{x-\min \left(x\right)}{\max \left(x\right)-\min \left(x\right)},$$

(17)

where $x^{\prime }$ is the new value from the formula, $x$ is the original value, and $\min \left(x\right)$ and $\max \left(x\right)$ are the minimum and maximum values of the column, respectively. The scaled data were input into the CNN for data feature extraction. The extracted features were input into the SE and LSTM. The SE designates weights to the input data in the CNN and classifies it as significant or non-significant. The LSTM estimates the power load using the data input into the CNN. The weighted values derived from the SE and power load estimation values outputted from the LSTM were input into the dense layer, resulting in the estimated power load of the ship. The estimated value was configured to allow the user to confirm the final value using inverse data scaling. Figure 4 shows the proposed model.

3.2. Model Training Process

The proposed CNN-LSTM with SE model was trained using the collected data, and classified into training, validation, and test datasets to prevent overfitting of the model. Figure 5 shows the ratio of the training, test, and validation datasets.

The data in the dataset were measured every 10 min. In this study, the power load in the next 10 min was forecast from the previous 50 min; thus, the time step was five. The CNN-LSTM with SE model uses the previously mentioned datasets to search for hyperparameters and estimate power values. The model forecasting steps are as follows:

Step 1.

Parameter initialization

The LSTM used in the proposed model used ReLU as the activation function.

Table 2. Initial parameters of the CNN-LSTM with the SE model.

Parameter Name	Initial Parameter Value
Number of first 1D convolution layer filter	10
Size of first 1D convolution layer kernel	3
Number of second 1D convolution layer filter	10
Size of second 1D convolution layer kernel	3
Number of neurons in excitation layer input	10
Number of neurons in excitation layer output	10
Number of neurons in first LSTM	16
Number of neurons in second LSTM	16
Number of neurons in third LSTM	16
Value of learning rate	10−6–10−3
Value of batch size	16

shows the initial parameter configurations of the 1D convolution, SE, and LSTM layers.

The parameters in Table 2 were optimized in the hyperparameter searching stage. Furthermore, the RMSE, mean absolute error (MAE), and mean absolute percentage error (MAPE) were used to measure the model performance, expressed as follows [34]:

$$RMSE=\sqrt{\frac{1}{N}{\sum }_{i=1}^N{\left(y_i-\widehat{y_l}\right)}^2},$$

(18)

$$MAE=\frac{1}{N}{\sum }_{i=1}^N\left|y_i-\widehat{y_l}\right|,$$

(19)

$$MAPE=\frac{1}{N}{\sum }_{i=1}^N\left|\frac{y_i-\widehat{y_l}}{y_i}\right|\times 100,$$

(20)

where $N$ refers to the total number of data utilizing the performance evaluation function, $y_i$ is the ith true value, and $\widehat{y_l}$ is the ith estimated value. The RMSE penalizes large residual values using the square function. The MAE checks the mean of the absolute error by comparing the estimated value with the actual value to examine the two values against the absolute scaler. Finally, the MAPE evaluates the uniform estimation error of the forecast model as a percentage.

Step 2.

Searching hyperparameters

Bayesian optimization was used in searching for hyperparameters. The forecasting value and loss were calculated using an initial parameter value. The kernel function of Bayesian optimization was used to determine the optimized hyperparameter value.

Table 3. Search space of the CNN-LSTM with the SE model.

Parameter Name	Search Space
Number of first 1D convolution layer filter	10–128
Size of first 1D convolution layer kernel	3–128
Number of second 1D convolution layer filter	10–128
Size of second 1D convolution layer kernel	3–128
Number of neurons in excitation layer input	10–1024
Number of neurons in excitation layer output	10–1024
Number of neurons in first LSTM	16–1024
Number of neurons in second LSTM	16–1024
Number of neurons in third LSTM	16–1024
Value of learning rate	10−6–10−3
Value of batch size	16–256

shows the search space of the hyperparameters.

Step 3.

Forecasting

The test datasets were used to evaluate the performance of the proposed model with the optimal hyperparameters.

4. Results and Discussion

This section compares the performance of the proposed model with that of the parallel and direct CNN-LSTM models, primarily used in estimating conventional time series datasets [35,36]. The selected models were chosen based on previous research regarding electricity forecasting [37]. The parameters for the proposed model were obtained using the Bayesian optimization algorithm. Figure 6 shows the convergence plot of the proposed model with the Bayesian optimization algorithm. It can be observed that the Bayesian optimization algorithm rapidly reached the convergence point and minimum point of the loss function in the ninth iteration. Using a convergence plot confirmed that the Bayesian optimization algorithm applied to the proposed model was successfully implemented.

Table 4. Selected optimized hyperparameters of the proposed CNN-LSTM with the SE model by Bayesian optimization.

Parameter Name	Selected Hyperparameter Value
Number of first 1D convolution layer filter	90
Size of first 1D convolution layer kernel	3
Number of second 1D convolution layer filter	128
Size of second 1D convolution layer kernel	3
Number of neurons in excitation layer input	1024
Number of neurons in excitation layer output	194
Number of neurons in first LSTM	512
Number of neurons in second LSTM	528
Number of neurons in third LSTM	194
Value of learning rate	0.001
Value of batch size	256

lists the values of the optimal hyperparameters of the proposed model derived through Bayesian optimization.

Figure 7 shows the structures of the parallel and direct CNN-LSTM models used for the comparative evaluation.

The comparative evaluation experiments were performed using Python, Scikit-Learn, and Tensorflow libraries. Each model used the same train, test, and validation dataset ratio, with the power load in the next 10 min forecasted from the previous 50 min as the model training process. Figure 8 shows the actual data (true) and forecasting results for all models. The results of all models used for comparative evaluation were consistent with the actual data. However, the detailed comparative evaluations of the models in Figure 9 shows that, on average, severe amplitudes of the forecasted value occurring through forecasting appeared in the CNN-LSTM (direct) model. In particular, amplitudes were generated more intensely after the heavy loading section; values with higher differences than the actual data values were output. Furthermore, the forecasting capability was relatively poor in areas below 800 kW. The forecasting performance of the parallel CNN-LSTM model was better than that of the direct CNN-LSTM model but inferior to that of the proposed model. The latter was demonstrated by the unstable loading changes detected between 1000 kW and 2000 kW. Finally, the proposed model performed forecasting by reflecting the characteristics of the actual data values more smoothly than the parallel and direct CNN-LSTM models.

Table 5. Performances of the proposed CNN-LSTM with SE, CNN-LSTM (parallel), and CNN-LSTM (direct) models.

Model	RMSE	MAE	MAPE
CNN-LSTM with SE	148.81	90.41	5.73
CNN-LSTM (parallel)	159.85	100.29	6.52
CNN-LSTM (direct)	202.78	134.43	8.83

summarizes the performances of all models evaluated using RMSE, MAE, and MAPE. The proposed model demonstrated the best output.

5. Conclusions

Research on artificial intelligence using ship data has been increasingly conducted in recent years. The LSTM model processes time series data smoothly, and the CNN model extracts data features. Conventional time series data estimation models use the LSTM and CNN models simultaneously. This study used the SE model to enhance the performance of existing time series data forecasting models. Furthermore, the hyperparameters ensuring the highest performance reduced the time spent on hyperparameter searching using Bayesian optimization algorithms. The primary conclusions are summarized as follows:

• Bayesian optimization is suitable for determining hyperparameters for the proposed CNN-LSTM with the SE model by combining loss functions and hyperparameters. Compared with traditional methods, such as grid and random search, this process can determine the parameters more rapidly.
• This study used the parallel and direct CNN-LSTM load estimation models to evaluate the performance of the proposed model. According to the comparative test results, the proposed model demonstrated the best ship power load forecasting performance.
• More data are required to enhance the estimation performance of the proposed model for intermittent heavy loading.

We recognize that the analytical ability to forecast a ship’s electricity feature is essential to enhance the ability of the forecasting model. As well, this model can help ship’s simulation to make algorithms for reducing fuel oil consumption. However, the ability for heavy load forecasting is limited. Thus, we will research a co-occurrence forecasting model for a ship’s electricity in a follow-up study.