Research on MASS Collision Avoidance in Complex Waters Based on Deep Reinforcement Learning

Abstract

The research on decision-making models of ship collision avoidance is confronted with numerous challenges. These challenges encompass inadequate consideration of complex factors, including but not limited to open water scenarios, the absence of static obstacle considerations, and insufficient attention given to avoiding collisions between manned ships and MASSs. A decision model for MASS collision avoidance is proposed to overcome these limitations by integrating the strengths of model-based and model-free methods in reinforcement learning. This model incorporates S-57 chart information, AIS data, and the Dyna framework to improve effectiveness. (1) When the MASS’s navigation task is known, a static navigation environment is built based on S-57 chart information, and the Voronoi diagram and improved A* algorithm are used to obtain the energy-saving optimal static path as the planned sea route. (2) Given the small main dimensions of an MASS, which is easily affected by wind and current factors, the motion model of an MASS is established based on the MMG model considering wind and current factors. At the same time, AIS data are used to extract the target ship (manned ship) data. (3) According to the characteristics of the actual navigation of ships at sea, the state space, action space, and reward function of the reinforcement learning algorithm are designed. The MASS collision avoidance decision model based on the Dyna-DQN model is established. Based on the DQN algorithm, the agent (MASS) and the environment interact continuously, and the actual interaction data generated are used for the iterative update of the collision avoidance strategy and the training of the environment model. Then, the environment model is used to generate a series of simulated empirical data to promote the iterative update of the strategy. Using the waters near the South China Sea as the research object for simulation verification, the navigation tasks are divided into three categories: only considering static obstacles, following the planned sea route considering static obstacles, and following the planned sea route considering both static and dynamic obstacles. The results show that through repeated simulation experiments, an MASS can complete the navigation task without colliding with static and dynamic obstacles. Therefore, the proposed method can be used in the intelligent collision avoidance module of MASSs and is an effective MASS collision avoidance method.

1. Introduction

1.1. Background

With the rapid development of economic globalization and the shipping industry, more than 82.5% of goods are transported by water. In recent years, the continuous progress of human science and technology has further promoted the development of ship science and technology, developing towards large-scale, specialized, high-speed, and unmanned ships. The safety of ships sailing at sea has also gradually attracted attention. The analysis conducted by EMSA during safety investigations [1] determined that from 2014 to 2020, at accident events or contributing factor levels, 89.5% of maritime safety accidents were related to human action. Therefore, with the development of the ship intelligence industry and to ensure that human factors will not affect the navigation safety of ships, the research and development of collision avoidance technology of intelligent ships/unmanned surface vehicles (USVs) have become a general trend. The International Maritime Organization (IMO) put forward a new work plan on Maritime Autonomous Surface Ships (MASSs) in 2017–2018 and formulated relevant conventions and regulations to solve a series of problems such as safety and environmental protection. MASSs is the term the IMO uses for ships that, to varying degrees, can operate in part or completely independent of human interaction [2,3]. MASS research has become a research hotspot in the international maritime field. The research, development, and adoption of MASSs are becoming the development trend in the shipbuilding industry. The shipping industry and relevant scientific research institutions have invested in relevant research on autonomous ships with different degrees of autonomy and different levels of intelligence.

The navigation environment of ships at sea is complex and changeable. There are static obstacles such as shorelines, islands, reefs, and sunken ships and unidentified dynamic obstacles. They are also interfered with by factors such as time-varying and uncertain winds and currents at sea. Accidents related to the nature of ship navigation, such as ship collisions, contacts, and groundings/strandings, occur occasionally. As shown in Figure 1 and Figure 2 below, according to the investigation results of maritime accidents of the European Maritime Safety Agency (EMSA) [1], over the period 2014–2020, accidents of navigational nature (collisions, contacts, and groundings/strandings) represented 43% of all occurrences related to the ship. These three types of accidents accounted for 13%, 17%, and 13% of the total number of accidents respectively. Therefore, the ship collision avoidance maneuver is crucial for the safe navigation of ships at sea. At present, the increasing traffic flow of ships at sea leads to increasingly crowded navigation channels, further increasing the risk of ship collision. Ship collision accidents usually cause casualties, property losses, and even more severe damage to the ecological environment. To ensure the autonomous and safe navigation of an MASS at sea, it is urgent to develop its autonomous intelligent collision avoidance technology under complex navigation conditions.

The development of unmanned ships is still at an early stage, and there will be a long coexistence between manned and unmanned ships in the future. However, the International Regulations for Preventing Collisions at Sea 1972 (COLREG) only aims at collision avoidance between manned ships. Therefore, the collision avoidance problem between manned and unmanned ships is an urgent problem to solve. Currently, unmanned ship/smart ship/MASS refer to the same object, only the name is different. Therefore, intelligent collision avoidance between MASSs and manned ships is an urgent problem.

1.2. The Literature Review

There are few studies on MASSs and manned ships and most focus on collision avoidance decision making between manned ships. According to the classification of research methods, ship collision avoidance models can be divided into traditional and intelligent ones.

The traditional ship collision avoidance model is based on traditional methods. The traditional methods include analytic geometry (AG), velocity obstacle (VO), fuzzy logic (FL), Swarm Intelligence Algorithm (SIA), and a mixture of these algorithms.

The application of the Marine Collision Avoidance System (MCAS) and Automatic Radar Plotting Aid (ARPA) system led to the subsequent proposal of the collision avoidance decision-making model based on AG. Wilson et al. [4] proposed a collaborative decision-making model of ship collision avoidance based on the line-of-sight method. Larson et al. [5,6] applied the Morphin algorithm to draw multiple arcs before the ship, covering the local obstacle map to consider all safe paths. Casalino et al. [7] proposed a local obstacle avoidance method based on the concept of the Bounding Box, where the Bounding Box is defined as a rectangular area that ships should avoid. Simetti et al. [8] introduced a safe Bounding Box around the original collision boundary box in the continuous workspace. Szlapczynski et al. [9] proposed a method for determining, organizing, and displaying collision avoidance information based on the Collision Threat Parameter Area (CTPA), which can improve the handling measures for MASSs in heavy weather conditions. Kim et al. [10] considered the COLREG. They proposed a collision avoidance algorithm based on the isochron method, including calculating collision risk and planning the optimal path for multi-vessel collision avoidance. Gail et al. [11] put forward the Collision Avoidance Dynamic Critical Area (CADCA) concept based on the minimum maneuvering area required by the two ships in the encounter. They explained the relevant hydrodynamic effects of ship dynamics and different rudder angles and forward speeds. Subsequently, Gail [12] improved the CADCA and proposed a collision avoidance model between ships and static obstacles.

The idea of the VO appeared in 1980 when it was named CTPA [13]. Later, Pederson et al. [14] applied it to the navigation field and proved that this method could provide better collision avoidance decision support for the officer on watch (OOW) compared with traditional ARPA equipment. Kuwata et al. (2014) [15] considered the COLREG and designed a ship local path planner based on the VO. Chen et al. [16] proposed a new risk detection method for ship collisions based on the VO and AIS data. Huang et al. [17] proposed a ship collision avoidance model based on the VO algorithm for collision avoidance scenarios with nonlinear motion characteristics and predictable target ship trajectories. Subsequently, this model was improved, and a maritime collision avoidance system for ships based on the Generalized VO (GVO) algorithm was proposed [18]. Li et al. [19] proposed a ship dynamic path planning method based on a multi-level Morphin adaptive search tree algorithm and VO. This method considers the ship’s maneuverability and the COLREG.

As a nonlinear control method independent of the controlled object, the fuzzy mathematics method can be applied well to the intelligent collision avoidance of ships. Liu et al. [20] combined the FL method with a neural network to quantify ship collision avoidance decisions. Perera et al. [21] considered the COLREGS and proposed an intelligent collision avoidance decision-generation system based on FL. Subsequently, Perera et al. [22] proposed a collision avoidance decision-generation model and execution of collision avoidance behavior based on FL theory and the Bayesian network. Ahn et al. [23] used the network-based adaptive fuzzy inference system (ANFIS), expert system, and multilayer perceptron (MLP) in the ship collision avoidance system.

SIO algorithms mainly include Ant Colony Optimization (ACO), Particle Swarm Optimization (PSO), and so forth. Tsou et al. [24] proposed a path-planning system based on the ACO after considering the COLREG and navigation practice. Lazarowska [25] proposed an ACO-based path planning method for USVs in the dynamic marine environment, which can be applied to the decision support system on board or the intelligent obstacle detection and collision avoidance system.

The intelligent collision avoidance model includes the model established using advanced artificial intelligence (AI) technology to study the decision-making problem of ship collision avoidance. AI technology includes Artificial Neural Networks (ANNs) and Reinforcement Learning (RL).

The ANN has been a research hotspot in AI since the 1980s. It abstracts the human brain neural network from the perspective of information processing, establishes a simple model, and forms different networks according to different connection methods. As early as the 1990s, some scholars successively used this method to establish decision-making models of ship collision avoidance. Simsir et al. [26] established a ship position prediction model using the ANN to solve the collision avoidance problem of ships in narrow waterways, laying a foundation for subsequent determination of the possibility of collision between two ships. Praczyk et al. [27] proposed an automatic multi-ship collision avoidance for system based on an evolutionary neural network to solve the collision avoidance task in complex, multi-objective, and rapidly changing environments. Xu et al. [28] proposed an automatic collision avoidance method based on a deep convolution neural network (CNN) using the solid visual processing ability of machine vision technology. Lin et al. [29] proposed a recursive neural network (RNN) with a convolution unit to improve the autonomy and intelligence of obstacle avoidance planning for unmanned underwater vehicles. This method uses a convolution layer to replace the entire connection layer in the standard RNN, thus reducing the number of parameters and improving the ability to extract features. Johansen et al. [30] used a Bayesian belief network from systems theoretical process analysis (STPA) to establish an online risk assessment model for ships. As an advanced method in the field of AI, RL has made remarkable achievements in the field of games and control. RL is suitable for solving decision-making problems with time sequence, which is consistent with the decision-making process of intelligent driving, so it is gradually applied to the intelligent decision making of USVs, unmanned aerial vehicles, and unmanned vehicles. RL can be divided into model-based RL and model-free RL according to whether the environment model is established. At present, model-free RL methods are primarily used in the field of intelligent collision avoidance, mainly including Q-learning, State Action Reward State Action (SARSA), Deep Q-Network (DQN), Policy Gradient (PG), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), and Actor Critical (AC) and its derivative algorithms. Yin et al. [31] proposed a simple obstacle avoidance algorithm based on Q-Networks to deal with complex navigation situations and unknown environmental dynamics. Shen et al. [32] put forward a multi-level intelligent collision avoidance algorithm for unmanned ships based on the DQN after fully considering the COLREGs, experience in maritime navigation, and ship maneuverability. Zhang et al. [33] proposed a MASS autonomous navigation decision-making method based on the DQN and artificial potential field. Woo et al. [34] considered the COLREG and proposed a collision avoidance model for unmanned vessels based on machine vision and the DQN. Wu et al. [35] proposed an autonomous navigation and intelligent collision avoidance algorithm based on the Dueling DQN (DDQN). Liu et al., [36] aiming at the problem that RL is time-consuming and infeasible in dealing with complex tasks, combined DQN with Transfer Learning (TL), which can transfer the knowledge learned in simple tasks to closely related but more complex tasks, providing a new idea for ship collision avoidance. Ryohei et al. [37] proposed an automatic collision avoidance system for ships based on grid sensors and the PPO, then improved this algorithm (Ryohei et al., 2020) [38], and proposed a multi-ship collision avoidance and waypoint navigation model based on the Long Short-Term Memory networks (LSTM) and the Proximal Policy Optimization (PPO). Zhao et al. [39] considered the COLREGs and the ship’s maneuverability and used DQN to directly map the ship’s status to the ship’s rudder angle command and used the PG-based AC algorithm to train the multi-ship collision avoidance model. Jiang et al. [40] considered COLREGs and proposed an autonomous ship collision avoidance method based on deep RL (DRL) and the attention mechanism. The method includes collision risk assessment and motion planning and analyzes the rationality and effectiveness of collision avoidance decisions from the perspective of collision risk and the nearest safe distance. Heiberg et al. [41] proposed a decision-making model of ship intelligent collision avoidance based on the DRL and the PPO. They applied the most advanced collision risk index to the reward design of the model.

Model-free RL methods can be applied to many fields. Because there is no need to establish a model, all agents’ decisions are obtained through interaction with the environment, so this method applies to scenes that are difficult to model or cannot be modeled at all. Similarly, without models, agents need to interact with and explore the environment constantly, requiring a lot of trial and error, leading to the most significant disadvantage of model-free RL: low data efficiency. This algorithm often needs to interact with the environment hundreds of thousands, millions, or even tens of millions of times. There are few intelligent collision avoidance models based on model-based RL. In order to solve the problem of low efficiency of model-free RL, Xie et al. [42] proposed a compound learning method based on the asynchronous advantage actor-critic (A3C) algorithm, LSTM, and Q-learning. This method combines the advantages of model-based and model-free RL. The author applies this method to multi-ship intelligent collision avoidance at sea, and the simulation results show that the decision-making performance of this method is superior to the standard A3C model and the traditional optimization model.

Through the analysis and comparison, it can be concluded that the main problems of the decision-making model of ship collision avoidance are: (1) They only consider static obstacles and do not consider the shape of obstacles or simplify their shapes to a circle. (2) To simplify the collision avoidance situation and the generation of collision avoidance decisions, the motion state of the own ship and the target ship is excessively simplified. For example, if the ship’s motion model is simplified or the motion factors are ignored, the collision avoidance measures taken by ships in the close encounter may fail easily. The ship’s dynamic model rarely considers the effects of wind and flow. (3) The interference of environmental factors needs to be fully considered, such as visibility, sea conditions, narrow or open waters, and the impact of traffic flow density at that time. (4) In the collision avoidance problem of MASSs, it is assumed that MASSs, like manned ships, should comply with the COLREGS [43,44]. However, MASSs differs significantly from manned ships regarding intelligence, main dimensions, maneuverability, and tasks performed [45]. (5) For the decision algorithm of ship collision avoidance based on the RL, the biggest problem of model-free RL is the low data efficiency. In contrast, the biggest challenge of the model-based RL method is the model’s error. (6) For the decision-making algorithm of ship collision avoidance based on the ANN, there is a problem with local optimization.

1.3. Research Content

In order to solve the above problems, this study refers to Gao et al.’s hypothesis on MASS maritime navigation rules [45] (due to the strong maneuverability of an MASS, its navigation should not interfere with the standard navigation of manned ships) and applies the previous studies on collision risk [46] and track prediction model [47]. A MASS collision avoidance decision-making model based on the Dyna-DQN is proposed, considering wind and current:

(1)

The static navigation environment is established using the S-57 chart and grid method, and the navigation safety weight of each grid is calculated. On the premise of known MASS navigation tasks, the optimal energy-saving static path is generated based on the Voronoi chart and improved A* algorithm [48].

(2)

The handling performance of an MASS is fully considered, taking into account its small main dimensions and that its maneuverability are easily affected by wind and current. The MASS motion model, under the influence of wind and current, is established based on the MMG model.

(3)

To make the navigation track of the target ship more realistic, the AIS track of a region near the South China Sea in January 2018 is extracted and cleaned. A specific track is selected as the navigation track for dynamic obstacles. As the AIS data are sent and received at unequal intervals, the AIS trajectory of the target ship is linearly interpolated here to match the trajectory points of the MASS. To better simulate the real navigation environment, ensure a full understanding of the target ship’s navigation trajectory, and better determine the potential risks and collision avoidance decisions, the online multiple outputs Least-Squares Support Vector Regression model based on selection mechanism (SM-OMLSSVR) [47] is used to predict the future trajectory of the target ship.

(4)

The RL model’s state space, behavior space, and reward function are designed. In the state space, the state of the own ship, the current and predicted state of the dynamic obstacle, the collision risk between two ships, and the environmental state are considered. The reward function considers whether to exceed the chart boundary, the extent of deviation from the planned route, whether to collide with static and dynamic obstacles, the extent of drift, and whether to reach the target point.

(5)

Based on the advantages of model-based and model-free RL, a decision-generation model of collision avoidance based on the Dyna DQN algorithm is established. In the model-based part, agents will continue to interact with the environment based on the DQN algorithm to generate real interactive data and use these data to optimize decision-making behavior and build a world model simultaneously. A virtual world model is used to generate virtual interactive data, and these data are used to optimize decision-making behavior.

(6)

To verify the effectiveness of the proposed model, three collision avoidance tasks are designed: only considering static obstacles; considering both static obstacles and planned sea route; and considering static obstacles, dynamic obstacles, and planned sea route. The experimental results show that the proposed model is an effective and efficient decision-making model of collision avoidance.

The remainder of this study is organized as follows. In Section 2, theoretical knowledge and concepts related to the model are described. In Section 3, the structure of the collision avoidance decision model between MASSs and manned ships based on Dyna-DQN is described. The verification and case study are carried out in Section 4. In Section 5, conclusions and future research are presented.

2. Methods and Tools

2.1. DQN Algorithm

Q-learning is a time-difference (TD) learning method used to estimate the state-action value function Q, and it is an off-policy method. When the environment is known and the states and actions are limited, an exhaustive search can be performed on all possible state-action pairs $\left(s,a\right)$ to find the optimal Q value ($Q^{\ast }$). If an environment has many states with multiple actions in each state, then traversing the entire environment will waste a lot of time. Therefore, it is better to find some parameter $\theta$ to approximate the Q function, namely:

$$Q\left(s,a;\theta \right)\approx Q^{\ast }\left(s,a\right),$$

(1)

The basic idea of the DQN algorithm comes from Q-Learning. However, its difference from Q-Learning is that its state-action value function Q is not directly calculated by state value $s$ and action $a$ but is calculated by the Q network. The input is the state $s$; the output is the state-action value function Q of all actions in this state. This Q network is the approximate Q function mentioned above. It is a neural network, which can be DNN, CNN, or RNN.

When nonlinear functions (such as neural networks) are used to represent the state-action value function Q, they are often unstable or even nonconvergent, which is caused by the strong correlation between data and does not meet the requirements of independent and identical distribution of neural networks. DQN takes the following two measures to solve this problem: (1) The method of Experience Replay (ER) is used to save the rewards and status updates obtained from each interaction between the agent and the environment. In the subsequent Q value update, only random data sampling is needed, which can eliminate the correlation in the observation sequence and smooth the changes in the data. (2) Using the iterative update, a new target network is added to calculate the target state-action value function $Q\prime$, and the target network is updated periodically to reduce the correlation of data.

The algorithm minimizes the loss function through adjustment $\theta$ and improves the state-action value function $Q\left(s,a;\theta \right)$. The loss function is shown as follows:

$$L\left(\theta \right)=E_{\left(s,a,r,s\prime \right)}\left[{\left(y_i-Q\left(s,a;\theta \right)\right)}^2\right],$$

(2)

$$y_i=r+\gamma \underset{a\prime }{\max }Q\prime \left(s\prime ,a\prime ;\theta \prime \right),$$

(3)

where $\gamma \in \left[0,1\right]$ is the discount factor, and $Q\prime \left(\cdot \right)$ is the target state-action value function. By differentiating $\theta$ in the loss function, we can obtain the following gradient:

$${\nabla }_{\theta }L\left(\theta \right)=E_{\left(s,a,r,s\prime \right)}\left[\left(r+\gamma \underset{a\prime }{\max }Q\prime \left(s\prime ,a\prime ;\theta \prime \right)-Q\left(s,a;\theta \right)\right){\nabla }_{\theta }Q\left(s,a;\theta \right)\right],$$

(4)

The pseudocode for DQN is shown in Algorithm 1. Among them, $\epsilon$ is the $\epsilon$ parameter in the $\epsilon$-greedy strategy. The $\epsilon$-greedy strategy is a common choice behavior strategy, which aims to balance the relationship between exploration and exploitation. It means that when the agent makes a decision, there is a small probability of a positive number $\epsilon$ to randomly select an unknown action, and the remaining probability of $1-\epsilon$ is to select the action with the greatest value in the existing action.

Algorithm 1: DQN

1: Initialize replay memory D to capacity N

2: Initialize action-value function Q with random weights $\theta$

3: Initialize target action-value function $Q\prime$ with weights $\theta \prime =\theta$

4: For episode = 1, M do

5:          Initialize the sequence of state $s$

6:          For $t=1$, T do

7:                    With probability $\epsilon$ select a random action $a_t$

8:                    otherwise select $a_t=\arg {\max }_{a}Q\left(s_t,a;\theta \right)$

9:                    Execute action $a_t$ in emulator and observe reward $r_t$ and $s_{t+1}$

10:                    Store transition $\left(s_t,a_t,r_t,s_{t+1}\right)$ in D

11:                    Sample random minibatch of transitions $\left(s_t,a_t,r_t,s_{t+1}\right)$ from D

12:                    Set $y_j=\left\{\begin{array}{l}{r}_j \mathrm{for} \mathrm{terminal} s_{j+1}\\ r_j+\gamma {\max }_{a\prime }Q\left(s_{j+1},a\prime ;\theta \right) \mathrm{for} \mathrm{non}-\mathrm{terminal} s_{j+1}\end{array}\right.$

13:                    Perform a gradient descent step on Equation (2) with respect to the network parameter $\theta$.

14:                    Every C steps reset $Q\prime =Q$

15:           End For

16: End For

2.2. Dyna Framework

RL can be divided into the model-based method and the model-free method. Here, the model represents the environmental dynamics characteristics constructed by individuals, which is called the virtual world model in this paper. The model-free RL method makes the agent interact with the environment directly to obtain data and uses the interactive data directly to optimize the agent’s action. The model-based RL method first learns the virtual world model from the data and then optimizes the strategy based on this model. Unlike model-free RL, which can only obtain values of other unknown states by trying to interact with the environment, model-based RL can use the virtual world model learned from the data to predict values of other unknown states. This method does not need to be tried too often and often has a strong generalization ability, which can significantly improve data utilization efficiency. However, model-based RL algorithms cannot be used if the system cannot be modeled, such as games or natural language processing. A model system, such as a ship, can be adapted to this approach. The ship’s motion conforms to the most fundamental laws of physics, and the system can be modeled using the developed kinematics and dynamics principles of rigid body and fluid. It significantly improves the data utilization rate, reduces the model’s training time, and promotes the real-time decision making of the ship. The biggest challenge encountered by this learning method is a model error; that is, the model learning through data has a model error. Especially at the beginning of training and with a small amount of data, the model learned is bound to be inaccurate. Using an inaccurate model make predictions will produce a significant error. Unlike model-based RL methods, model-free RL algorithms have an excellent property: asymptotic convergence. The model-free RL algorithm can guarantee the optimal solution for the agent after countless interactions with the environment. However, because the model-free RL method does not fit the model, the agent can only perceive and recognize the environment through continuous interaction with the environment, which is enormous, tens of thousands or even millions of times. So many interactions make model-free RL algorithms inefficient and difficult to apply to the physical world.

In order to solve the problem of model-based and model-free RL, full use is made of the advantages of the two. When they are combined, the Dyna framework emerges: the model-free RL method is used to learn the virtual world model from experience, whereas the experience and the virtual experience based on the virtual world model sampling are used to learn, plan, and update the value or strategy function, as shown in Figure 3. The Dyna-Q algorithm’s flow based on state-action value is shown in Algorithm 2.

Algorithm 2: Dyna-Q

1: Initialise $Q\left(s,a\right)$ and $M\left(s,a\right)$ for all $s\in S$ and $a\in A\left(s\right)$

2: Do forever:

3:          $s\leftarrow$ current (nonterminal) state

4:           $a\leftarrow \epsilon -greedy\left(S,Q\right)$

5:          Execute action a; observe resultant reward $r$, state $s\prime$

6:           $Q\left(s,a\right)\leftarrow Q\left(s,a\right)+\alpha \left[r+\gamma {\max }_{a}Q\left(s\prime ,a\prime \right)-Q\left(s,a\right)\right]$

7:          $M\left(s,a\right)\leftarrow s,a,r,s\prime$ (assuming deterministic environment)

8:            Repeat $n$ times:

9:                    $s\leftarrow$ random previously observed state

10:                   $a\leftarrow$ random action previously taken in $s$

11:                   $r,s\prime \leftarrow M\left(s,a\right)$

12:                     $Q\left(s,a\right)\leftarrow Q\left(s,a\right)+\alpha \left[r+\gamma {\max }_{a}Q\left(s\prime ,a\prime \right)-Q\left(s,a\right)\right]$

2.3. Dyna-DQN

The Q-learning algorithm only applies to the situation where the state space and the action space are discrete distributions, and both spaces are small. Because the state space of ships is relatively large and continuous, to better apply the RL algorithm to the collision avoidance decision making of ships, the model-free part of the Dyna-Q Algorithm is replaced by the Q-learning algorithm with the DQN algorithm; therefore, the Dyna-DQN model is formed. The pseudo code of this model is shown in Algorithm 3. It can be seen that the algorithm is divided into three parts: (1) based on the model-free RL, the agent interacts with the environment based on the DQN algorithm to collect real empirical data and improve the strategy; (2) learning of the virtual world model, using real experience to learn and improve this model, which includes two classification tasks of $s\prime$ and $done$ and a regression task $r$; and (3) using the virtual world model for planning. The agent uses the empirical data obtained from this model to improve the strategy.

Algorithm 3: Dyna-DQN

Require:$N, \epsilon , K, L, C, Z$

Ensure:$Q\left(s,a;{\theta }_Q\right)$

1: Initialize $Q\left(s,a;{\theta }_Q\right)$ and $M\left(s,a;{\theta }_M\right)$

2: Initialize $Q^{\prime }\left(s,a;{\theta }_{Q^{\prime }}\right)$ with ${\theta }_{Q^{\prime }}={\theta }_Q$

3: Initialize real experience reply buffer $D^u$ sing Reply Buffer Spiking (RBS), and simulated experience reply buffer $D^s$ as empty

4: for n = 1:N do

5:        # Direct RL starts

6:        generate an initial state s

7:        while s is not a terminal state do

8:              with probability $\epsilon$ select a random action a

9:              otherwise select $a=\arg {\max }_{a\prime } Q\left(s,a\prime ;{\theta }_Q\right)$

10:              execute a, and observe reward r and the state of the next moment $s\prime$

11:              store $\left(s,a,r,s^{\prime }\right)$ to $D^u$

12:                 $s=s^{\prime }$

13:        end while

14:       sample random minibatches of training samples $\left(s,a,r,s^{\prime }\right)$ from $D^u$

15:       update ${\theta }_Q$ via Z-step minibatch Q-learning according to Equation (2)

16:        # Direct RL ends

17:        # World Model Learning starts

18:       sample random minibatches of training samples $\left(s,a,r,s^{\prime }\right)$ from $D^u$

19:       update ${\theta }_M$ via Z-step minibatch SGD of multi-task learning

20:        # World Model Learning ends

21:        # Planning starts

22:        for k = 1:K do

23:               done = FALSE, l = 0

24:               generate an initial state s

25:              while done is FALSE $\cap$ $l\le L$ do

26:                     with probability $\epsilon$ select a random action a

27:                     otherwise select $a=\arg {\max }_{a\prime }Q\left(s,a\prime ;{\theta }_Q\right)$

28:                     execute $a$

29:                     world model responds with r and the state of the next moment $s\prime$

30:                     store $\left(s,a,r,s^{\prime }\right)$ to $D^s$

31:                      $l=l+1$$, s=s\prime$

32:               end while

33:              sample random minibatches of training samples $\left(s,a,r,s^{\prime }\right)$ from $D^s$

34:              update ${\theta }_Q$ via Z-step minibatch according to Equation (4)

35:        end for

36:        # Plannning ends

37:       every C steps reset ${\theta }_{Q\prime }={\theta }_Q$

38: end for

2.4. MMG Model Considering Wind and Flow Factors

The Manoeuvring Model Group (MMG) model, also known as the separated model, is an internationally popular mathematical model of ship motion [49].

The MMG model can simulate the ship’s motion with six degrees of freedom at most, but the research of pitching, rolling, and heaving motion is of little significance in the study of ship collision risk in open water, good weather, and sea conditions. Therefore, only the ship’s swaying, surging, and yawing motions are studied here, and the influence of wind and current is taken into account. According to the basic idea of the MMG model, the external force and external torque acting on the hull are decomposed into the fluid force of the bare hull, propeller force, rudder force, etc. When the origin of the attached coordinate system is the ship’s center of gravity, the details are as follows:

$$\left\{\begin{array}{l}\left(m+m_x\right)\dot{\mathit{u}}-\left(m+m_y\right)\mathit{v}\mathit{r}={\mathit{X}}_{\mathit{H}}+{\mathit{X}}_{\mathit{P}}+{\mathit{X}}_{\mathit{R}}\\ \left(m+m_y\right)\dot{\mathit{v}}+\left(m+m_x\right)\mathit{u}\mathit{r}={\mathit{Y}}_{\mathit{H}}+{\mathit{Y}}_{\mathit{P}}+{\mathit{Y}}_{\mathit{R}}\\ \left({\mathit{I}}_{ZZ}+{\mathit{J}}_{ZZ}\right)\dot{\mathit{r}}={\mathit{N}}_{\mathit{H}}+{\mathit{N}}_{\mathit{P}}+{\mathit{N}}_{\mathit{R}}-{\mathit{Y}}_{\mathit{H}}{\mathit{x}}_c\end{array}\right.,$$

(5)

where ${\mathit{X}}_i$, ${\mathit{Y}}_i$, and ${\mathit{N}}_i$ $\left(i=\mathit{H},\mathit{P},\mathit{R}\right)$ are the transverse, longitudinal, and yaw forces and moments of the bare hull, propeller, and rudder, respectively, acting on the ship; ${\mathit{I}}_{ZZ}$ and ${\mathit{J}}_{ZZ}$ are the moment of inertia of yawing and additional moment of inertia, respectively; $\mathit{u}$, $\mathit{v}$, and $\mathit{r}$ are the speed along the $x$-axis, the $y$-axis, and the rotation speed of yawing, respectively; $\dot{\mathit{u}}$, $\dot{\mathit{v}}$, and $\dot{\mathit{r}}$ are the acceleration along the $x$-axis, $y$-axis, and rotation acceleration of yawing, respectively; and $x_c$ is the transverse coordinate of the center of the ship.

Considering the influence of fluid on ship motion under uniform current, the velocity of current $V_e$ is decomposed into the $G_x$-axis and $G_y$-axis of the ship’s moving coordinate system:

$$\left\{\begin{array}{l}{u}_c=V_e\cos \left(C_f-C\right)\\ v_c=V_e\sin \left(C_f-C\right)\end{array}\right.,$$

(6)

where $u_c$ is the component of the current’s velocity on the $G_x$-axis; $v_c$ is the component of the current’s velocity on the $G_y$-axis; $C_f$ is the angle of current (calculated clockwise from true north of 0°, consistent with the direction of flow in navigation); and $C$ is the course of the ship.

Then, the components $u_1$ and $v_1$ of the speed over ground (after considering the flow) on the $G_x$-axis and the $G_y$-axis are expressed as:

$$\left\{\begin{array}{l}{u}_1=u+u_c\\ v_1=v+v_c\end{array}\right..$$

(7)

As for the ship’s yaw angular velocity, it remains the same whether on water or on the ground. The relation between accelerations can be obtained by differentiating the time in Equation (7):

$$\left\{\begin{array}{l}\dot{u}={\dot{u}}_1-r{v}_c\\ \dot{v}={\dot{v}}_1+r{u}_c\end{array}\right..$$

(8)

The force of the wind on a ship is mainly related to the superstructure and its layout, the wind direction, and the wind speed. The relationship is as follows (the wind pressure remains unchanged):

$$\left\{\begin{array}{l}{X}_W=\frac{1}{2}{\rho }_a{A}_r{V}_W^2{C}_{WX}\left({\theta }_r\right)\\ Y_W=\frac{1}{2}{\rho }_a{A}_L{V}_W^2{C}_{WY}\left({\theta }_r\right)\\ N_W=\frac{1}{2}{\rho }_a{A}_L{L}_{oa}{V}_W^2{C}_{WN}\left({\theta }_r\right)\end{array}\right.,$$

(9)

where $X_W$, $Y_W$, and $N_W$ are longitudinal wind pressure, transverse wind pressure, and yaw moment of wind, respectively; ${\rho }_a$ is the air density; $A_r$ and $A_L$ are the forward projected area and the side projected area of the hull above the waterline, respectively; $V_W$ and ${\theta }_r$ are relative wind speed and relative windward angle, respectively; and $C_{WX}$, $C_{WY}$, and $C_{WN}$ are longitudinal wind pressure coefficient, transverse wind pressure coefficient. and torque coefficient, respectively. Their values can be obtained from wind tunnel tests or Isherwood’s regression equation if no experimental data are available.

2.5. Transformation of Coordinates

The ship position point data in AIS equipment is the latitude and longitude coordinates in the WGS-84 geodetic coordinate system. In order to accurately calculate the basic elements, such as azimuths and distances between two ships, the latitude and longitude coordinates need to be converted into Mercator coordinates under the Mercator projection. Mercator projection, also known as “positive axis isometric cylindrical projection”, is based on the characteristics of “isometric” to ensure that the shape of the projection object does not deform and further ensures the correctness of the orientation and mutual position, so it is often used in the field of navigation and aviation. Assuming that the longitude and latitude coordinates of the ship position point are $\left(\lambda ,\phi \right)$, the Mercator coordinates under the Mercator projection are $\left(x,y\right)$, and the formula for converting the longitude and latitude coordinates into Mercator projection coordinates is as follows:

$$\left\{\begin{array}{l}{r}_0=a\cos \phi /\sqrt{1-e^2{\sin }^2\phi }\\ q=\ln \tan \left(\frac{\pi }{2}+\frac{\phi }{4}\right)-\frac{e}{2}\ln \frac{1+e\sin \phi }{1-e\sin \phi }\\ x=r_0\times \lambda \\ y=r_0\times q\end{array}\right.,$$

(10)

where $r_0$ is the radius of latitude circle of the reference latitude, $a$ is the radius of the major axis of the earth ellipse, $q$ is the isometric latitude, and $e$ is the first eccentricity of the Earth ellipsoid.

3. Intelligent Decision-Making Model of Ship Collision Avoidance Based on the Dyna-DQN

Based on the S-57 chart data, AIS data, the Dyna-DQN model, and the MMG model, the decision model of collision avoidance based on the RL is designed as shown in Figure 4: (1) First, the shoreline and water depth data are extracted from the S-57 electronic chart when the navigation task is known. The Voronoi diagram and improved A* algorithm are used to obtain the optimal static path for energy saving [48]. (2) To ensure that the MASS will not reencounter static obstacles during its dynamic collision avoidance, the static environment extracted from S-57 is divided into two-dimensional plane grids. The grids containing obstacles are set as non-navigable areas, and the other are navigable. The navigable area grids are given corresponding safe navigation weights based on certain principles. (3) Because the main dimensions of an MASS are small, its maneuverability is easily affected by wind and current. In order to ensure the accurate modeling of MASS motion, based on the MMG model and using the wind and current data from the National centers for environmental prediction (NCEP) climate forecast system version 2 (CFSv2), the MASS motion model considering the impact of wind and current is established. (4) The AIS data of a particular area are used to extract the target ship’s trajectory and obtain the navigation trajectory of the target ship (manned ship). At the same time, in order to better simulate the real-time collision avoidance at sea and make forward-looking collision avoidance decisions, the ship track prediction model based on the SM-OMLSSVR algorithm [47] is used to predict the target ship’s navigation track for some time in the future. (5) The RL model’s state space, action space, and reward function are designed. The state space is divided into three aspects: the own ship, the target ship, and the navigation environment. The state space of the target ship includes the current position point of the target ship and the predicted future position point. The action space is the difference between the own ship’s course that can be safely navigated in this encounter situation and the course of the previous moment. The reward function can be divided into six aspects: exceeding chart boundaries, tracking paths, reaching the goal, collision with static obstacles, collision with dynamic obstacles, and drift. (6) The MASS collision avoidance decision model based on the DQN-Dyna model is established. The DQN algorithm enables the agent (ship) to interact with the environment continuously. The interaction data generated are used for the iterative update of the collision avoidance strategy and the training of the environment model simultaneously. Then, a series of simulated empirical data is generated using the virtual world model to promote the iterative update of the strategy. Finally, from a global perspective, based on the environment of S-57 chart rendering, state space, action space, and reward function, integrating the ship’s handling characteristics, taking into account the static obstacles, dynamic obstacles, and the COLREG, the MASS intelligent collision avoidance decision-making model based on the Dyna-DQN and the MASS collision resolution module are built.

3.1. Assumptions and Conceptual Definitions

3.1.1. Assumptions

According to the maritime navigation regulations for MASSs formulated by Gao et al. [35], due to the strong maneuverability of MASSs, their avoidance priority should be lower than that of manned ships. That is, when a MASS is navigating at sea and meets a manned ship, the passage of a MASS shall not interfere with the normal passage of any other manned ship. Therefore, we make the following assumptions:

(1)

No matter what kind of encounter situation the MASS and the target ship (manned ship) are in when there is a collision risk between the MASS and the target ship (manned ship), it is assumed that the MASS takes the action of collision avoidance. In contrast, the target ship (manned ship) keeps course and speed. The target ship (manned ship) here includes power-driven vessels, vessels not under command, vessels restricted in her ability to maneuver, vessels engaged in fishing, and sailing vessels.

(2)

Due to the strong manipulability of the MASS and its good communication performance, the MASS does not have to comply with Rule16 of the COLREGS, that is, “Every vessel which is directed to keep out of the way of another vessel shall, so far as possible, take early and substantial action to keep well clear.”

3.1.2. Conceptual Definitions

(1)

Complex waters: usually refers to waters with poor natural conditions, complex ship traffic flow, and great difficulty in navigation, including offshore construction waters, multi-channel intersection waters, channel bend sections, narrow channel sections, shoal navigation areas, reef waters, etc.

(2)

Virtual world model: a model trained with accurate interactive data based on a particular method. In the model, when the state sequence and action variables are input, the corresponding next-time state sequence, reward value, and whether the termination state value is reached will be output.

3.2. Construction of Environment

3.2.1. Analysis and Extraction of S-57 Electronic Chart Data

When an MASS navigates at sea, radar, camera, and other sensors cannot provide global navigation environment information. In addition, an MASS cannot directly use the electronic chart to avoid collision independently, so it is necessary to study the S-57 data structure of the electronic vector chart. Through analyzing the electronic chart file, the marine geographic information and the marine environment information required by the MASS for collision avoidance decision making are extracted, and irrelevant information is deleted. The chart information is rendered into an information pattern that the MASS can recognize.

The electronic chart comprises sea area elements such as submarine terrain, navigation obstacles, signs, port facility, etc. The standard library format of the S-57 electronic chart is ISO/IEC8211 international standard. Based on the ISO8211 open-source library, this paper analyzes its electronic chart information according to the library structure of the S-57 chart. The format of the S-57 electronic chart standard library is shown in Figure 5 below. It selects the regions with longitude and latitude ranges of 112.50° E–113° E and 21.50° N–21.98° N. Rendering the S-57 chart of this area and the extracted chart information is shown in Figure 6 below.

3.2.2. Generation of Optimal Static Path

Using the previous research on the MASS’s global and energy-saving path planning, based on the consideration of water depth, tide, wind, and current factors, the Voronoi diagram and the improved A* algorithm are used to generate the optimal energy-saving static path for the MASS. Assuming that the starting point and end point coordinates of the MASS are (112°36′ E, 21°44′ N) and (112°55′ E, 21°40′ N), respectively, the optimal static path generated is shown in the following Figure 7.

3.2.3. Grid the Static Environment

When there is no dynamic obstacle in the environment, the navigation environment model of the MASS can be regarded as its arbitrary movement in the finite area of two-dimensional sea level, where the finite area is the navigable area without static obstacles. Because the number of static obstacles on the sea is limited, the shape and distribution are uncertain. In order to ensure that the MASS will choose to avoid obstacles in the navigable area during the subsequent dynamic obstacle avoidance, this paper will divide the original environment extracted from the S-57 chart into several grids of equal size through grid division. Determining whether there are static obstacles (such as land, island, and shoal) parsed from the electronic chart in the grid, in turn, divides the grid map of the environment into the navigable area and non-navigable area.

In addition, the MASS may deviate from the planned route when avoiding collisions or performing tasks in navigable grids close to non-navigable grids. However, if there are no obstacles in the grids, there is still a specific potential risk of navigation. Therefore, this paper referred to [50] to set navigation safety weights to guide the MASS not to enter potential risk areas as much as possible. It is assumed that the navigation safety weight of the non-navigable grid is 0, and the navigation safety weight $w\left(C_i,R_i\right)$ of the navigable grid $\left(C_i,R_i\right)$ will be affected by the number of non-navigable grids in the eight adjacent grids. The specific relationship is as follows:

$$w\left(C_i,R_i\right)=1-\frac{1}{8}n,$$

(11)

Figure 8 below shows the environment map after grid division, in which the darkest grid is a non-navigable grid, and the rest are navigable grids. The smaller the navigation safety weight of the navigable grid, the darker the color. For more transparent observation grid processing, the black box area in the figure is enlarged to produce the image on the right, where the darkest shadow area is the unnavigable area, and the rest are navigable areas. The depth of the shadow color in the navigable area represents the navigation safety weight, and the darker the color, the lower the navigation safety weight of the grid.

3.2.4. Adding Wind and Current Data

The study area selected in this paper is the sea area near the South China Sea, with longitude and latitude ranges of 112.5° E−113° E and 21.5° N−21.98° N, respectively. The wind data are from the second version of the climate prediction system of the National Center for Environmental Prediction (NCEP), and the wind data from January 2018 in the study area are selected. The current data come from the global Hybrid Coordinate Ocean Model (HYCOM) and the Navy Coupled Ocean Data Assimilation (NCODA) analysis, and the time update frequency is 3 h. The current data from January 2018 in the study area are selected. It should be noted that the selected area is small, resulting in less wind and current data in this area. To solve this problem, the Kriging interpolation method [51] is used to interpolate the spatial points’ wind and flow data. This method is a regression algorithm for spatial modeling and prediction (interpolation) of random process/random field based on covariance function. It is a typical geostatistical algorithm widely used in geographic science, environmental science, atmospheric science, and other fields. The interpolated wind and current data in this area are shown in Figure 9. Because this paper only uses the wind data in the offshore area, the wind data in the land area are not visualized here.

3.2.5. Extraction and Prediction of the Target Ship’s Trajectory

The AIS data of this water area from 1 January to 7 January 2018 are taken and visualized, as shown in Figure 10. As the AIS data are sent by ship-based AIS equipment according to specific rules and unequal intervals based on the alteration of course, speed, and other factors, the update interval of Class A AIS information is shown in

Table 1. Update interval of Class A AIS Information.

Ship Status	Report Interval
Anchor/berthing vessel, speed < 3 knots	3 min 1
Anchor/berthing vessel, speed > 3 knots	10 s 1
Speed < 14 knots	10 s 1
Speed > 14 knots and change course	$3\frac{1}{3}$ s 1
Speed is 14–23 knots	6 s 1
Speed is 14–23 knots and change course	2 s
Speed > 23 knots	2 s
Speed > 23 knots and change course	2 s

¹ When the shipboard AIS is confirmed as a synchronous sign station, its dynamic in-formation update interval is 2 s.

below. In order to better set the collision scenario where the MASS and the target ship are in danger of collision, the selected target ship AIS data are interpolated at equal intervals in this paper. Figure 11 shows an AIS track of a ship with the Maritime Mobile Service Identity (MMSI) of 412000212 in the water area. The red curve in this figure is the AIS original track, and the blue dot is obtained by interpolating the original track points at equal intervals. At the same time, to make the generated collision avoidance decision have a certain forward-looking element, the ship track prediction model based on the SM-OMLSSVR [47] can predict the future trajectory of the target ship.

3.3. Setting of State Space, Action Space, and Reward Function of Collision Avoidance Decision Model Based on the Dyna-DQN

3.3.1. State Space

The states in the state space are divided into three categories: the state space of the own ship ${\mathit{S}}_{\mathrm{OS}}$, the state space of the target ship ${\mathit{S}}_{\mathrm{TS}}$, and the environmental information ${\mathit{S}}_{\mathrm{env}}$. Among them, the environmental information ${\mathit{S}}_{\mathrm{env}}$ comprises the navigation safety weights of 24 grids near the grid where the ship is located, as shown in Figure 12. If all 24 grids are navigable squares, ${\mathit{S}}_{\mathrm{env}}$ is composed of 24 ones; if all the grids are not navigable, ${\mathit{S}}_{\mathrm{env}}$ is composed of 24 zeros. When the ship is at the edge of the chart, there may be no grid around it; therefore, it is outside the boundary of the chart. At this time, the navigation safety weight of the corresponding position is supplemented by 0, indicating that the area is not navigable.

$${\mathit{S}}_{\mathrm{OS}}=\left({\mathit{x}}_o,{\mathit{y}}_o,C_o,{\delta }_o,{\tilde{\phi }}_o,C_e,y_e,{\psi }_e,‖{\mathit{P}}_{goal}-{\mathit{P}}_o‖,t\right),$$

(12)

$${\mathit{S}}_{\mathrm{TS}}=\left({\mathit{x}}_t,y_t,C_t,V_t,‖{\mathit{P}}_o-{\mathit{P}}_t‖,‖C_o-C_t‖,R_t,{\mathit{x}}_{{}_p}^1,y_{{}_p}^1,{\mathit{x}}_{{}_p}^2,y_{{}_p}^2,{\mathit{x}}_{{}_p}^3,y_{{}_p}^3\right),$$

(13)

$${\mathit{S}}_{\mathrm{env}}=\left(w_0,w_1,w_2,\dots ,w_{23}\right),$$

(14)

where ${\mathit{x}}_o$, ${\mathit{y}}_o$ and ${\mathit{x}}_{{}_t}^i$, ${\mathit{y}}_{{}_t}^i$ are the positions of the own ship and the i-th target ship, respectively; $C_o$ and $C_t$ are the course of the own ship and the target ship, respectively; $‖{\mathit{P}}_{goal}-{\mathit{P}}_o‖$ is the distance between the current position of the own ship and the destination; $‖{\mathit{P}}_o-{\mathit{P}}_t‖$ is the distance between the own ship and the target ship; $‖C_o-C_t‖$ is the heading intersection angle of the own ship and the target ship; ${\delta }_o$ is the rudder angle of the own ship; $y_e$ is the cross error, i.e., the distance between the own ship’s position and the planned sea route, as shown in Figure 13; ${\psi }_e$ is the bow error of the own ship, which is used to characterize the influence of wind and current on the ship motion model; $C_e$ is heading deviation; and ${\tilde{\phi }}_o$ is the relative angle between the own ship’s course and the own ship’s point to the destination, that is, the included angle from the own ship’s course line to the line between the own ship’s current position and the destination. The clockwise sign is positive, and the counterclockwise sign is negative. $R_{o,t}$ is the risk of collision between the own ship and the target ship [46]; $t$ is the operation time of the own ship; ${\mathit{x}}_{{}_p}^1, {\mathit{y}}_{{}_p}^1, {\mathit{x}}_{{}_p}^2, {\mathit{y}}_{{}_p}^2, {\mathit{x}}_{{}_p}^3, {\mathit{y}}_{{}_p}^3$ is the track point of the target ship at the next three moments obtained from the ship track prediction model using the SM-OMLSSVR [47]; and $w_0,w_1,w_2,\dots ,w_{23}$ are the navigation safety weights of 24 grids around the own ship.

3.3.2. Action Space

Because the main dimensions of the MASS are small and the maneuverability is more robust than that of general large-scale manned ships, it is assumed here that the MASS does not have to comply with Rule16 of the COLREGS, that is, “Every vessel which is directed to keep out of the way of another vessel shall, so far as possible, take early and substantial action to keep well clear.”

The course difference $\Delta C$ is taken as the output parameter of this model, that is, the parameter of the action space. The range of course difference is set as $\Delta C\in \left[-10°,10°\right]$ and discretized into:

$$\Delta C\in \left[-10°,-8°,-6°,-4°,-2°,0°,2°,4°,6°,8°,10°\right].$$

(15)

3.3.3. Reward Function

• Out chart boundaries

$$R_{\mathrm{out}}=\left\{\begin{array}{l}-r_{out} ,\mathrm{if} \mathrm{the} \mathrm{boundary} \mathrm{of} \mathrm{the} \mathrm{chart} \mathrm{is} \mathrm{exceeded}\\ 0 ,\mathrm{else}\end{array}\right..$$

(16)

• Tracking paths

$$R_{path}=-r_{path}{y}_e.$$

(17)

• Reaching the goal

$$R_{\mathrm{goal}}=\left\{\begin{array}{l}{r}_{\mathrm{goal}} \mathrm{if} \left|P_t-P_{\mathrm{goal}}\right|<5 \\ -r_{\mathrm{goal}}\left(\left|P_t-P_{\mathrm{goal}}\right|-\left|P_{t-1}-P_{\mathrm{goal}}\right|\right) \mathrm{otherwise}\end{array}\right..$$

(18)

• Collision with dynamic obstacles

$$R_{dcol}=-r_{dcol}{R}_{o,t}.$$

(19)

• Drift

$$R_{\mathrm{drift}}=\left\{\begin{array}{l}-r_{\mathrm{crosserr}} \mathrm{if} \left|u\right|<\left|v\right|\\ 0 \mathrm{otherwise}\end{array}\right..$$

(20)

• Collision with static obstacles

$$R_{scol}=\left\{\begin{array}{l}-r_{{}_{scol}} , \mathrm{if} w\left(C_i,R_i\right)=0\\ -r_{{}_{scol}}\left(9-\left(\begin{array}{l}w\left(C_{i-1},R_{i-1}\right)+w\left(C_i,R_{i-1}\right)+w\left(C_{i+1},R_{i-1}\right)\\ +w\left(C_{i-1},R_i\right)+w\left(C_i,R_i\right)+w\left(C_{i+1},R_i\right)+\\ w\left(C_{i-1},R_{i+1}\right)+w\left(C_i,R_{i+1}\right)+w\left(C_{i+1},R_{i+1}\right)\end{array}\right)\right),\mathrm{otherwise}\end{array}\right..$$

(21)

• The total reward in the model is as follows:

$$R_{total}=R_{out}+R_{path}+R_{\mathrm{goal}}+R_{dcol}+R_{\mathrm{drift}}+R_{scol},$$

(22)

where $r_{out}$, $r_{path}$, $r_{goal}$, $r_{dcol}$, $r_{drift}$, and $r_{scol}$ are the weight parameters of six awards. The MASS has two navigation modes during navigation: (1) Autonomous navigation mode. When the distance between two ships is significant, the collision risk $R_{o,t}$ is 0; that is, the dynamic obstacle collision risk reward cannot be considered at this time. (2) Collision avoidance mode. When two ships have potential collision risk, navigation task and collision avoidance risk shall be considered simultaneously.

4. Simulation Experiment

Under the machine learning framework of Pytorch developed by Facebook-AI-research (FAIR), the Dyna-DQN model is implemented using Python. A desktop computer with an i7 3.00 GHz CPU, 16 GB RAM, and NVIDIA TITAN RTX GPU was used to train the network.

In order to verify the effectiveness of the algorithm proposed in this paper, three scenarios are considered, namely, autonomous navigation of ships under static obstacles only, autonomous navigation of ships along planned routes under static obstacles only, and autonomous navigation of ships along planned routes under both static and dynamic obstacles. At the same time, the algorithm is compared with the DQN algorithm to verify its efficiency. The basic settings of the parameters are as follows: the learning rate of DQN and the neural network in the virtual world model is 10⁻³, the batch size in the DQN is set to 4000, the memory capacity is set to 10,000, and the discount factor $\gamma$ is set to 0.88. During the training process, the value $\epsilon$ decreases linearly, with the initial value set at 1.0 and the minimum value set at 0.01. The purpose is to make the proportion of exploration in the initial stage of the navigation task larger. With training, the proportion of exploration decreases, and the proportion of exploitation increases, that is, the $\epsilon$ linear decreases.

4.1. Autonomous Navigation of Ships with Static Obstacles Only

The autonomous navigation of ships with static obstacles only refers to the navigation of MASSs along the planned route with static obstacles considered. When the navigation task of the ship has been determined, that is, the starting point and destination of the navigation have been determined, it is assumed that the speed of the own ship during the whole navigation process remains unchanged, and the speed is 12.35 knots. Only by altering the course to avoid static obstacles is the scene under the task, as shown in Figure 6 above. Currently, both ${\omega }_{dcol}$ and ${\omega }_{path}$ in Formula (1) are set to 0. The training is divided into three rounds every 100 times.

Table 2. Only considering the training results under static obstacles.

	Total Episodes	Out of the Boundary	Collision with the Static Obstacle	Out of Time	Number of Successes
Run 1	100	10	88	2	0
Run 2	100	12	86	1	1
Run 3	100	7	91	0	2

shows the times of exceeding the boundary, touching static obstacles, exceeding the time threshold, and success in three training rounds.

Because the training environment is relatively complex, the training track with the return value $R_{\mathrm{total}}\ge -2000$ will not be displayed to visualize the training process more clearly. The area surrounded by black curves is static, such as land or islands and reefs. The area where the small red flag is located is the destination. Here, we assume that the MASS navigates to a distance of 500 m from the destination to reach the destination. At the same time, the color of the track being trained will be set to pink, the color of the track that has been trained and with $R_{\mathrm{total}}\ge -2000$ but has not reached the endpoint will be set to blue, and the color of the track that has successfully reached the endpoint will be set to bright red. The training results are shown in Figure 14. As seen from this figure, due to the complex navigation environment, the training effect is not good at Run1. With increased training times, the ship can reach its destination, and the success rate continues to increase.

4.2. Training of Ships Sailing According to the Planned Sea Route under the Condition of Only Considering Static Obstacles

In the case of only considering static obstacles, the autonomous navigation of the own ship is trained according to the planned sea route in Figure 8. The training goal is that the own ship will navigate along the planned sea route from the starting point to the destination. Compared with Section 4.1, the subtask of following the planned sea route is added. The following Figure 15, Figure 16 and Figure 17 show the training results, in which the bold black curve is the planned sea route of the MASS. The planned sea route is the optimal energy-saving path of the MASS generated using the Voronoi diagram and improved A* algorithm considering water depth, tide, wind, current, and other factors. The training is divided into three rounds, 500 times per round. The training results are shown in

Table 3. The training results when both static and dynamic obstacles are considered.

	Total Episodes	Out of the Boundary	Collision with Static Obstacles	Out of Time	Number of Successes
Run 1	200	23	177	0	0
	300	31	260	4	5
	500	38	443	12	7
Run 2	200	16	184	0	0
	300	22	278	0	0
	500	40	456	2	2
Run 3	200	18	177	4	1
	300	36	256	6	3
	500	56	425	15	4

.

Similarly, the training environment is more complex due to the addition of the sub-task of following the planned sea route. In order to present the training results more clearly, only the training track meeting the condition $R_{\mathrm{total}}\ge -2500$ is displayed for visualization. The color settings are the same as those in Section 4.1. Figure 15 is the schematic diagram of the visualization of the training results in the first round. Due to the numerous and disordered tracks that failed in training, in Figure 16 and Figure 17, the visual conditions were adjusted to display only the training tracks that met $R_{\mathrm{total}}\ge -2000$, and the color settings were changed. The areas surrounded by the blue curve were static obstacles such as land, islands, and reefs. The bold pink broken line was the planned sea route designed under the known navigation task, and $R_{\mathrm{total}}\ge -2000$ and the tracks that did not reach the destination were displayed with black curves. The track showing the successful arrival at the destination is displayed with the bold red curve. It can be seen from Figure 15, Figure 16 and Figure 17 that with the increase in training times, the training success rate continues to increase.

4.3. Collision Avoidance of Ships Considering Both Static and Dynamic Obstacles

Figure 18 shows the situation of two ships encountering in consideration of a static environment. The red curve is the ship’s planned sea route, and the blue curve is the extracted AIS track of the target ship. It is assumed that the ship and the target ship will meet at point B and collide when their respective course and speed remain unchanged. According to assumption (1), because the target ship is a manned ship and the own ship is an MASS, the own ship should take collision avoidance actions so as not to hinder the safe navigation of the target ship (manned ship). The training is divided into four rounds. Because this scenario considers dynamic obstacles and the conditions are more complex, the number of training times per round is set to 1500, and the training results are shown in

Table 4. The training results when both static and dynamic obstacles are considered.

	Total Episodes	Out of the Boundary	Collision with Static Obstacles	Collision with Ship	Out of Time	Number of Successes
Run 1	500	11	465	16	7	2
	1000	22	936	33	8	2
	1500	47	1390	49	10	5
Run 2	500	9	470	16	3	2
	1000	26	943	19	9	3
	1500	56	1382	39	18	5
Run 3	500	6	489	4	0	1
	1000	30	1246	11	5	8
	1500	42	1415	24	42	11

. The specific training situation is shown in Figure 19, Figure 20 and Figure 21. The curve settings in these figures are consistent with Figure 16 and Figure 17, except that the green curve represents the extracted target ship track. It can be seen from these figures that with the increase in training times, the success rate gradually increases.

4.4. Comparison Experiment with Decision Model of Ship Collision Avoidance Based on DQN Algorithm

In order to prove the superiority of the model proposed in this paper, under the condition of considering both static obstacles and planned sea routes as well as dynamic obstacles, it is compared with the decision-making model of ship collision avoidance based on the DQN algorithm. The setting of the parameters in the DQN algorithm is the same as in the Dyna-DQN. The two models are trained three times, each time for 3000 episodes. The average reward values for the two models under three training times are shown in Figure 22 and Figure 23.

As shown in Figure 22 and Figure 23, the average reward value obtained using the Dyna-DQN algorithm fluctuates within [–3500, –7500] and eventually converges around −5000. The average reward value obtained by the DQN algorithm always fluctuates within [−6000, −8000]. At the 3000 episodes, the average reward value returns to near the initial reward value again. It can be seen that under 3000 episodes of training, compared to the decision-making model of ship collision avoidance based on the DQN, the initial and late average reward values obtained by the decision-making model of ship collision avoidance based on the Dyna-DQN are relatively large, and the overall trend is upward. It was found through a complex and thorough search that the DQN is a model-free reinforcement learning method that requires a large amount of interactive data for strategy update, resulting in low data utilization and a huge training time. The Dyna-DQN algorithm uses interactive data to establish a virtual world model. It simultaneously uses interactive data and virtual data generated by the virtual world model to update strategies, improving data utilization and thus improving training efficiency. Although the initial stage has a small amount of data and overlapping training results, in the later stage, as the number of episodes increases, the model accuracy gradually improves; the average reward value of the decision-making model of ship collision avoidance based on the Dyna-DQN shows an overall upward trend.

5. Conclusions and Future Studies

5.1. Conclusions

This paper considers environmental factors such as wind and current, static obstacles, dynamic obstacles, and the ship’s maneuverability. Moreover, it combines the advantages of model-based RL and model-free RL and designs a decision model of ship collision avoidance based on the Dyna-DQN. In order to verify the effect of this model, three levels of training tasks were designed: MASS autonomous navigation considering only static obstacles, MASS navigation according to the planned sea route considering only static obstacles, and MASS autonomous navigation considering both static and dynamic obstacles. Through repeated trial and error training, it can be concluded that an MASS can reach the destination without colliding with obstacles. With increased training times, the task’s success rate gradually increases. At the same time, to prove this model’s superiority, the traditional DDQN model is used for comparison. The test results show that the proposed model can converge faster, and the total reward value is higher than the DDQN model.

The differences between this study and other studies are as follows:

(1)

In the current research on ship collision avoidance decisions, most studies are set in open waters and do not consider static obstacles. This paper selects S-57 chart information to build a static navigation environment and calculates the navigation safety weight of each grid through grid processing to ensure that static obstacles will not be touched when performing collision avoidance actions.

(2)

The problem of ship maneuverability is not considered in the existing collision avoidance decision-making research. This study reflects that the main dimensions of an MASS are relatively small and it is vulnerable to wind and current, and an MMG motion model considering the influence of wind and current factors is established.

(3)

The existing research assumes that MASSs should comply with the COLREGs when avoiding collision and does not consider the collision between MASSs and manned ships. Given the strong maneuverability of an MASS, this study determined that an MASS should not interfere with the normal navigation of the target ship (manned ship).

(4)

Currently, most collision avoidance decision models based on RL use the model-free RL methods. The sample utilization rate is low, and the training time is long. This paper combines the model-based RL method and model-free RL to establish a Dyna framework, which improves the sample utilization rate and reduces the training time.

To conclude, the model has great application potential and can be applied to the autonomous navigation of MASSs and the intelligent collision avoidance between MASSs and manned ships. It is an efficient MASS autonomous navigation and collision avoidance model.

5.2. Future Studies

Although the model presented in this paper has shown good training results in the simulation test, there are still some problems that need to be supplemented and improved in future research, as follows:

(1)

Considering the problem of time cost, the study only considers collision avoidance of two ships and does not consider collision avoidance decisions involving three or more ships. The next step is to expand the number of target ships gradually.

(2)

Considering the increase of training complexity caused by the increase of the dimension of collision avoidance behavior, only altering is considered in collision avoidance behavior. The next step is to consider adding speed to the action space.

(3)

Currently, the model belongs to the simulation test stage, and the next step is to carry out the sea trial and use the model on the unmanned ship.

(4)

Currently, the frontier of RL includes imitation RL, reverse RL, meta RL, hierarchical DRL, multi-task transfer DRL, and DRL based on memory and reasoning. In the future, we should consider whether these methods can be better applied to unmanned ship intelligent collision avoidance from theoretical and practical perspectives.

(5)

The collision avoidance study between MASSs and manned ships is carried out under “MASS should avoid manned ships”. The next step should also consider the collision avoidance of manned ships under the condition of complying with COLREGs and the collision avoidance between two MASSs.