Semantic Modeling of Ship Behavior in Cognitive Space

Abstract

Ship behavior is the semantic expression of corresponding trajectory in spatial-temporal space. The intelligent identification of ship behavior is critical for safety supervision in the waterborne transport. In particular, the complicated behavior reflects the long-term intentions of a ship, but it is challenging to recognize it automatically for computers without a proper understanding. For this purpose, this study provides a method to model the behavior for computers from the perspective of knowledge modeling that is explainable. Based on our previous work, a semantic model for ship behavior representation is given considering the multi-scale features of ship behavior in cognitive space. Firstly, the multi-scale features of ship behavior are analyzed in spatial-temporal dimension and semantic dimension individually. Then, a method for multi-scale behaviors modeling from the perspective of semantics is determined, which divides the behavior scale into four sub-scales in cognitive space, considering spatial and temporal dimensions: action, activity, process, and event. Furthermore, an ontology model is introduced to construct the multi-scale semantic model for ship behavior, where behaviors with different semantic scales are expressed using the functions of ontology from a microscopic perspective to a macroscopic perspective consecutively. To validate the model, a case study is conducted in which ship behavior with different scales occurred in port water areas. Typical behaviors, which include leveraging the axioms expression and semantic web rule language (SWRL) of the ontology, are then deduced using a reasoner, such as Pellet. The results show that the model is reasonable and feasible to represent multi-scale ship behavior in various scenarios and provides the potential to construct a smart supervision network for maritime authorities.

1. Introduction

There is a high traffic density in some busy waterways, especially in port areas, where some severe situations have occurred. It increases the supervision difficulty to vessels for maritime authorities, such as the Maritime Safety Committee (MSC) and services. Specifically, the supervision to vessels includes static information inquiry, tracking of one or more vessels, ship behavior recognition, etc. The fact facilitates the autonomous supervision to vessels, especially whose behaviors in congestion areas are riskier than in normal areas. The rapid development of Maritime Autonomous Ships (MASS) in recent years has also placed a demand on the autonomous recognition and semantic transformation of ship behavior, which MASS should ideally satisfy to improve the perception of surrounding ship behavior. As a result, more and more researchers are paying attention to the automatic recognition and semantic enrichment of ship behavior.

Ship behavior is the representation of the trajectories of ships enriched with various types of semantic attributes. It is challenging to recognize ship behavior without any instructions from the human aspect for computers. In particular, complicated behaviors comprise a set of simple behaviors enriched with geographic attributes, temporal features, motion characteristics, etc. For example, the behavior of anchoring implies the place where the behavior occurred (anchorage), the time duration of the behavior (long-term), and the motion state (stationary). These behaviors are commonly used in realistic scenarios currently and require a deeper study. It is challenging to clarify all of the behaviors by computers without a proper model in which the semantic features of behaviors can be considered in depth. By contrast, a human expert can quickly and precisely understand exactly ship behavior. This is due to the excellent capability of processing information collected from multiple sources in a cognitive space for humans. Such a capability is what is required for the intelligent computers of MSC or MASS.

The semantic modeling of behaviors in cognitive space is a process of semantic reflection of the movement of physical objects, which enables computers to understand behaviors in the same way that humans think. Hence, it is a feasible way to empower a computer to be capable of recognizing behaviors enriched with rich semantics. However, there are a wide variety of behaviors with different semantics, as stated above. It is impractical to program each behavior manually. Thus, a model to extract and collate the semantic characteristics of ship behavior is desired to be provided to reach the final goal of semantic modeling.

This work is based on the previous work [1,2], focusing on modeling and reasoning of semantic ship behavior with different scales in multiple dimensions. We propose a semantic model to extract and recognize multi-scale behaviors automatically in cognitive space based on historical automatic identification system (AIS) data. In this study, the features of ship behavior that represent corresponding trajectories are analyzed from the perspective of spatial-temporal and semantic, respectively. Furthermore, a multi-scale semantic model is given to depict ship behavior in cognitive space, in which behaviors with different spatial scales are sorted out and a formalized cognitive model of behavior is presented. Moreover, by means of the ontology modeling method, multi-scale behaviors are explored and expressed further. Behaviors with different semantic scales are presented, leveraging the functions of ontology. Finally, a case study of a ship approaching into and leaving a port is given to show how the model works.

The remainder of the paper is structured as follows. In Section 2, an overview of related work is given. The analysis of the multi-dimensional feature is in Section 3. The model of multi-scale behaviors is proposed in Section 4. Section 5 constructs an ontology model for multi-scale semantic behaviors. Section 6 presents a case study to validate the feasibility of the semantic model. In Section 7, the results and discussion of the experiment are presented. Finally, the conclusion is given in Section 8.

2. Related Work

2.1. Cognitive Modeling

There is extensive literature on the topic of cognition modeling for human behaviors, which are influential in ship behavior modeling in cognitive space. A number of studies have examined the construction method of knowledge base [3,4,5,6], and knowledge reasoning [7] with ontology [8,9,10], which discusses cognitive modeling and knowledge reasoning for human activities. [4,6,8] constructed the ontology based on the relationships between humans and then environment to transform human behaviors from the data layer to the semantic level, which realized the recognition of human behavior intelligently. These studies are beneficial to provide some thoughts about how to construct the cognitive framework of ship behavior.

Currently, there are few works that focus on the research of semantic recognition of ship behavior. [9] designed a method to identify the ship events using AIS data that records relevant information about ship movement, such as position, speed, course, etc. [10] tries to deduce the complicated behaviors based on the method proposed in [9], providing the foundation of cognition modeling of complicated ship behavior.

2.2. Semantic Expressions of Trajectory

To address the problem of semantic behaviors recognition for computers, from the perspective of humans, some recent studies centered on semantic modeling from a human perspective have proposed semantic computational processing methods [11], using the same approach that humans use to perform semantic recognition of behaviors. Refs. [12,13] proposed a semantic computing frame to compute the trajectory generated by moving objects, such as vehicles, humans, and animals. Based on these efforts, some studies on ship semantic behaviors have also been carried out. A semantic model of ship behavior was proposed in [14], which takes into account the uncertainty of the occurring behavior; [15] mined the pattern of ship trajectories by means of semantic annotation and possibility modeling; and [1] constructed the ontology model of ship behavior, considering the temporal relationships between each other. In addition, some projects work on the semantic computing of trajectories in the maritime domain. An example of this is datAcron [10,16,17], a project focusing on the representation of semantic trajectories of aviation and maritime conceptualizations.

2.3. Ship Behavior Modeling

In order to model ship behavior explicitly, many studies have been focusing on behavior modeling from trajectory to behavior. There are two kinds of methods used to analyze it: probabilistic statistics and motion characteristics extracting and modeling. The former refers to the pattern mining of ship behavior by means of statistics analysis [18,19,20]. Another approach based on motion characteristics analysis accomplishes this by analyzing the relationship between the characteristics and behaviors and then modeling. Ref. [21] considered the motion characteristics of ship trajectories to construct the model. Ref. [14] proposed a model for ship behavior based the ship basic behaviors, such as turning to port side, turning to starboard side, and some semantic behavior occurred in the environment.

Few studies on ship behavior undertake basic behavior modeling and prediction, considering the structural and temporal features of complicated behaviors, which are necessary for computers to satisfy the requirements to ascertain a desirable understanding of behaviors. Ref. [2] proposed a framework for ship behavior from a cognitive and semantic modeling perspective and constructed a semantic model to represent behavior from data to trajectory to complex behavior, considering its motion data and environmental attributes.

2.4. Multi-Scale Modeling of Trajectory

There are extensive studies focusing on the topic of multi-scale characteristics analysis in geography [22,23], which have explored in detail about the multi-scale characteristics of spatial-temporal objects [24,25,26]. The trajectory, as the representation of the spatial-temporal characteristics of physical objects, exhibits multi-scale characteristics. Previous studies provide a benchmark of multi-scale feature analysis for spatial objects and a solid basis for building a cognitive framework for modeling multi-scale ship behavior. Ref. [27] discussed the multi-scale representation of battlefield situation. Ref. [28] proposed a multi-level model to explore the spatial-temporal patterns of crime in different spatial scales of area. They provide guidance for the construction of cognitive models of ship behavior.

It is necessary to propose a systematic approach to analyze complicated behaviors by comprehensively considering its various characteristics, such as motion characteristics, topological relationships with environmental entities, etc. In general, the modeling of complicated behaviors needs to be considered in different dimensions.

As the semantic representation of ship trajectory, the multi-scale features of ship trajectory can form the multi-scale features of behaviors in three dimensions, such as time, space, and semantics. However, few studies have considered the semantic multi-scale features of ship behavior that are crucial for behavior recognition. The relationship of ship behavior between different levels and between different scales of the same level has not been constructed properly, which limits the development of the modular computing capability of the autonomous system for the safety supervision of behavior.

To address the problem, Firstly, we characterize ship behavior from the scale of spatial-temporal and analyze the shortcomings of modeling ship behavior in this dimension.

Secondly, we analyze the way ship managers with different cognitive mindsets perceive ship behavior and propose a cognitive model for ship behavior from the semantic dimension, dividing ship behavior into four layers of action, activity, process, and event to describe ship behavior at different spatial-temporal and semantic scales. Finally, the cognitive ontology of ship behavior is constructed, taking the typical behavior of ships in port areas as an example for ontological modeling and expression, and exploring the mechanism of multi-scale semantic expression and reasoning of ship behavior in port waters.

3. Multi-Dimensional Characterization of Ship Behavior in Cognitive Space

A ship generates a series of trajectory segments driven by the intention of the seafarer. That means that the semantics implied by the trajectory reflects the seafarer intention to navigate. From simple behaviors, such as accelerating and going straight, to advanced behaviors are the semantics implied by a ship’s trajectory, such as sailing along the fairway, berthing, etc. In other words, the behavior can be represented as the semantic reflection implied by the trajectories produced by physical objects in cognitive space where human operators process information on their own temporal and logical terms. That is, ship behavior has additional semantic features in addition to the spatial-temporal motion characteristics of ship trajectories. The semantics implied by trajectories are described differently within different spatial-temporal dimensions.

3.1. Previous Work for Semantic Modeling of Ship Behavior

For semantic modeling of ship behavior, we have explored in our previous studies [2], where a framework of semantic behavior generation process from trajectories enriched with motion semantics and topological environment semantics was given. In this paper, we proposed several concepts, such as atomic trajectory, atomic behavior, topological behavior, as well as traffic behavior, representing the semantic behavior with corresponding semantic features.

Specifically, we first divide the trajectory, generated from AIS data, into atomic trajectories, as trajectory units on the basis of our classification of atomic behavior. Atomic behavior represents the behavior of maintaining a constant motion state of both speed and course simultaneously, as shown in Figure 1. That means the trajectory was segmented according to its motion status instead of sample frequency or spatial grid division with same size, which is beneficial to reduce its computation complexity. Topological semantic enrichment is based on the atomic trajectory.

Following this, in order to enrich the semantics of geographical properties for trajectory unit, we introduced 15 spatial relations for the calculation of two objects involving point, line, and surface in maritime domain by adapting Dimensionally Extended 9-Intersection Model (DE-9IM) that is proposed for describing spatial relations of two regions.

Finally, we presented how traffic behavior, as semantic behavior unit of ships, including motion status and topological semantics, are formed through atomic trajectory, atomic behavior, topological behavior, and environment. Traffic behavior $T_l^{\mathrm{r}}$ can be represented as Formula (1) performing as a sentence, where $\underset{¯}{\underset{¯}{T_i^{\mathrm{a}}}}$ represents atomic trajectory as the subject, $\underset{¯}{T_j^{\mathrm{o}}}$ represents topological behavior as the predicate, $\underset{\sim \sim \sim }{r_k}$ represents the environment as the object, and $\left[B_i^{\mathrm{a}}\right]$ refers to the atomic behavior as the gerund of the sentence, respectively.

$$T_l^{\mathrm{r}}=\left\{\underset{¯}{\underset{¯}{T_i^{\mathrm{a}}}}\cap \underset{¯}{T_j^{\mathrm{o}}}\cap \underset{\sim \sim \sim }{r_k}\cap \left[B_i^{\mathrm{a}}\right]\right\}$$

(1)

This model provides a way to reach the goal of semantic unit formation, supporting further high-level semantic modeling for complicated behaviors, which can be represented with a set of traffic behaviors. In addition, we explored the temporal relations preliminarily within complicated behaviors in [1], where we expected to depict complicated behaviors through combining simple behaviors.

Previous works present how to enrich semantics from different respects to trajectories, but there is a lack of extensive analysis on complicated behaviors, especially the relationships between different dimensions. On the basis of these work, we try to propose an extended semantic model for complicated behaviors combining human cognitive habits.

3.2. Multi-Dimensional Feature Analysis

Based on previous research, we expect to investigate how complicated behaviors can be represented in terms of basic semantic behaviors. Considering the intrinsic spatial-temporal and semantic scale features [29] of complex behaviors, we wish to propose a framework for the analysis of complex behaviors that considers spatial, temporal, and semantic dimensions. Thus, we analyze behavior in three dimensions.

In terms of the spatial-temporal dimension, ship trajectories as a form of spatial-temporal representation generated by physical objects, the determination of the spatial-temporal scale depends on the frequency with which the trajectories are sampled [30]. Therefore, the sampling frequency and granularity of ship trajectories must be determined when analyzing and modeling ship trajectories at multiple scales purely from the spatial-temporal dimension.

However, it is challenging to provide a standard method to determine the scale of the spatial-temporal dimension. Because people with different roles have different concerns about ship behavior, that is not appropriate. Therefore, the modeling of multi-scale features of the track also needs to be reworked around different needs for attention, which presents a higher standard and challenge for the accurate sampling of ship tracks. For example, mariners are more concerned with short-term vessel behavior, such as analyzing whether the target vessel around her is performing the maneuvers specified in COLREG. In contrast, VTS officers are more inclined to obtain a longer range or time interval of behavior, such as analyzing whether vessels within their jurisdiction are engaging in illegal activities. In other words, different people have different scales of attention to the behavior of vessels, involving differences in scale not only in the spatial-temporal dimension but also in the semantic dimension.

Therefore, the analysis of ship behavior should combine the spatial-temporal dimension with the semantic dimension. From a semantic point of view, when modeling ship behavior at multiple scales, we need to describe the behavior semantically in the spatial-temporal dimension at the same time. They need to obtain a good understanding of behavior by dividing the semantic space into several appropriate semantic scales, which are closer to the human habit of perceiving behavior.

4. Multi-Scale Cognitive Modeling of Ship Behavior from Semantic Dimension

Spatial-temporal data are prevalent with multi-level, multi-grain, and multi-resolution characteristics, and the analysis and extraction of these features is a prerequisite for their awareness and modeling. In addition, the model construction based on these features is also in line with the human cognitive habits of multi-dimensional and multi-features of spatial-temporal data. Therefore, for the spatial-temporal trajectories corresponding to ship behavior, we need to consider these multidimensional features mentioned above and consider the intrinsic relationship of each dimension and the relationship between them. In view of the cognitive habits of people with different roles in the maritime domain for ship behavior, ship behavior can be analyzed and modeled from microscopic scale to macroscopic scale.

4.1. Formalized Cognitive Expression of Ship Behavior

Behavioral cognition is the result of multifaceted description and expression of ship trajectory. Based on the analysis of cognitive elements, the cognitive expression of ship behavior, Cog, should be considered as a cognitive set, including four elements: who, what, when, and where, which can be expressed as Equation (2).

$$Cog=\left\{o,b,t,p\right\}$$

(2)

where o denotes the object where the behavior occurs; b is the behavior that occurs at the object; t represents the time, including instant and interval; and p is the place where the behavior occurs.

Considering the multi-scale characteristics of spatial-temporal trajectories, this paper divides the cognition of ship behavior into four layers in the cognitive space: action, activity, process, and event, according to the expression habit of ship behavior in the semantic dimension. The division of behavior cognition is based on two aspects, including motion features and the topological features.

4.2. Multi-Scale Division of Ship Behavior in Cognitive Space

4.2.1. Action

Considering the practical needs of users for ship behavior, when describing and calculating the microscopic behavior of a ship, this paper avoids the situation that causes the redundancy of successive division of equal time interval or equal distance interval trajectories and the complexity of calculating topological relations. In this paper, from the perspective of behavior semantics, the concept of action is introduced to represent the cognition results of the micro-semantic behavior features, which is to represent the behavior that the ship motion characteristic, involving both speed and course, remains unchanged during the current behavior stage, such as keep course and deceleration (KC_DE), turn left and deceleration (TL_DE), turn right and deceleration (TR_DE), etc. Action behavior is a behavior to characterize the basic motion characteristics of the trajectory without additional semantic information related to environment. The behavior enriched with rich semantic can be formed based the action behavior.

4.2.2. Activity

Activity is the cognition results of the behavior represented by the trajectory of action behavior, enriched with topological and geographical semantics, which represents the behavior based on the topological interaction and geographical semantic enrichment. The behavior of the activity occurs on the trajectory of action, which interacts with the entities in the environment, such as anchorage, berth, etc., which is the basic semantic unit of ship behavior, and complex semantic behavior can be expressed by the combination of a set of consecutive activity behaviors.

The behavior difference between the activity and the action is that the action only reflects the semantic of motion characteristics of the trajectory and do not include the semantics of the interaction between the trajectory and its surroundings. In contrast, the activity has more semantics than the action but also involves the semantics of spatial topological calculations and geographic semantic enrichment performed by the action trajectory with environmental entities.

4.2.3. Process

A process represents the behavior of a ship in which the spatial topological relationship between its trajectory and environmental entities remains constant while keeping its speed or course unchanged. That is, when the characteristics of the speed or course of ships change or the spatial topological relationship between its trajectory and environmental entities changes, the current process behavior turns to the next process behavior.

In contrast to activity behaviors, process behaviors are the extension of activity behaviors, which describe the interaction behaviors that occur between trajectories with constant speed and constant course and entities in geographic space, respectively. For example, the behavior of anchoring preparing can be regarded as a process behavior, which is usually accompanied by a series of action behaviors of deceleration, while the trajectory of this behavior keeps the same spatial topology relationship with the geographic entity of anchorage during this process until the behavior of deceleration is stopped, at which point the action behavior of the ship changes, which also means that the activity of the ship changes. Therefore, the process behavior of anchoring preparing indicates that a process behavior occurs for the ship, and its connection with the next process behavior is made through the activity behavior of beginning to anchor, and the moment when the act of beginning to anchor begins is the moment when the behavior of anchoring preparing ends and the moment when the next process behavior begins.

Similarly, when the spatial topological relationship between the ship trajectory and the geographic entity changes, it means that the ship experiences an activity behavior, which indicates the beginning of the next process behavior.

A process behavior tends to have a larger temporal and spatial scope than an activity behavior, and it represents that the ship is executing a certain task, such as the process behavior of anchoring represents that a ship is anchoring at anchor, and the activity of anchoring does not change during this period. In contrast to the activity behavior, the process behavior does not consider the change or not of the combined motion characteristics of speed and course, and its focus is on the change of spatial and topological state between the trajectory that remains constant either on the speed or the course and the surrounding environment.

The behavior of activity, which can be considered as one of the components of the behavior of process, is a trigger element condition between different processes and serves to trigger the end of the previous process and the beginning of the next one.

4.2.4. Event

Event behaviors, which represent the overall behavior of the vessel occurring in the current observation view or macroscopic behavior relative to the current reference target, describe the logical and temporal relationships between the behaviors. For example, the entire behavior of berthing and unberthing can be considered as one event, including the three process behaviors of decelerating for preparing to berth, berthing, and accelerating for unberthing. Event behavior can also denote the behavior that occurs in a larger temporal and spatial context, which is extensible. For example, the behavior of a ship sailing from one port to another can be regarded as a whole event containing several sub-events, including the departure event from port A, several subsequent events, and the arrival event at port B.

The event behavior corresponding to the macro behavior is a semantic aggregation of activities and processes or, alternatively, can represent a semantic aggregation of multiple events with related relationships. As can be seen from Figure 2, the goal of transforming from trajectories to multi-scale spatial-temporal semantic behaviors can be achieved and multi-dimensional modeling and representation of behaviors is realized.

5. Ontology Modeling for Ship Behavior

Based on the analysis of multi-scale behavior, this paper proposes an ontology model for ship semantic behavior modeling in a cognitive manner. The model introduces several concepts in cognition to model the cognitive framework of ship behavior, involving ship, behavior, time, and place, which supports a systematic interpretation of ship behavior by a computer, as shown in Figure 3. Figure 4 presents the ontology of ship behavior created according to our cognitive framework of ship behavior. Figure 4a, Figure 4b and Figure 4c show the class, object properties, and data properties interfaces in the ontology software Protégé, respectively.

5.1. Classes

• Ship

The ship is the generator of trajectory and the occurrence object of behavior. Therefore, it is necessary to identify the object of trajectory in behavior cognition. As a unique code for the ship, the number of the Maritime Mobile Service Identity (MMSI) can be used to determine the object that generates the behavior.

• Behavior

The class of behavior is one of core classes of the cognitive ontology. Based on previous work [2], ship semantic behaviors are divided into four categories, including atomic behaviors, topological behaviors, traffic behaviors, and advanced semantic behaviors. For a better understanding, The class of atomic behavior proposed in [2] are extended to refined divisions, including first-order and second-order atomic behaviors. The second-order atomic behavior corresponds to action, including 10 categories, while the first-order atomic behavior corresponds to the process where either the speed or the course keeps ships maintaining constant, as can be seen in Figure 2.

As the unit of semantic behavior, the traffic behavior is the basic element to describe advanced semantic behaviors, corresponding to the behavior of activity.

The advanced semantic behavior refers to high-level behaviors, such as the behavior of process and event, that can be formed combining multiple consecutive sort of traffic behaviors in specific application scenarios.

• Environment

The class of environment represents the set of surrounding spatial objects existing in the form of physical or virtual entities, such as anchorages, channels, control areas, and infrastructure.

• Time

In order to represent the temporal relationship of behaviors, we introduce an existed time ontology to our work, which is available online: http://www.w3.org/2006/time# (accessed on 27 September 2020), that includes both instant and interval that can fulfill the temporal functions, which is beneficial to describe complex behaviors that are temporal and logical. Specifically, the class of instant is to describe the transient state of behavior, and time interval represents the time quantum, which lasts for a certain period, including start time and end time to express the duration of behaviors.

According to the theory of interval algebra proposed by Allen [31], basic time relationship includes before, after, and equal. Furthermore, 10 types of relationships between instant and interval can be depicted with 3 kinds of basic relationships. It is the temporal and logic features of ship behavior that can be depicted in this way.

Besides the abovementioned, we connect them with their relationships to link this network. There are three kinds of arrows for that, including solid arrow, dashed arrow, and solid arrow with empty end. As for solid arrow, it refers to the relationship between two classes either same classes or different classes. For example, there is the relationship of “occur object” that has the domain—“traffic behavior” and the range—“ship”. For the second one, it means what the own data properties the entity have. An example can be taken to illustrate that the dashed arrow pointing to “Atomic Trajectory” from “Position” represents that the former one has the data of the latter one. The final one means there is the relationship of parent–subclass relationship, such as the arrows of the three behaviors in the top yellow round box pointing to “Advanced Semantic Behavior”.

Now that we have extracted the different classes, the next step is to connect them to form knowledge graph. For that reason, we then need to add the relationship properties with each other to them.

5.2. Property-Constrained Axiom

• The class of Atomic Behavior

Atomic behavior can be recognized and annotated by pre-processing and calculation of trajectories. Therefore, we import atomic behavior as instances of ship behavior directly into the ontology via its interface. On this basis, first-order atomic behavior can be expressed by second-order atomic behavior. For example, several instances can be illustrated using the property-constrained axiom as follows.

acceleration = (KC_AC or TL_AC or TR_AC)

keepSpeed = (KC_KS or TL_KS or TR_KS)

stop= stay

Furthermore, since the move behavior is one of the general behaviors, including all the first-order atomic behaviors, we likewise use the axiomatic expression of the property constraint to define the move behavior, which indicates that the ship is in the move state, including all the second-order atomic behaviors, which can be expressed as follows.

move = (reflects value KC_AC) or (reflects value KC_DE) or (reflects value KC_KS) or (reflects value TL_AC) or (reflects value TL_DE) or (reflects value TL_KS) or (reflects value TR_AC) or (reflects value TR_DE) or (reflects value TR_KS)

• The class of Topological Behavior

According to definition and semantic computing results for topological behavior, each topological behavior can be expressed with ontology via axiomatic expression method of attribute constraint. For example, Topo1 represents the topological relationships between trajectory and navigation environment, which can be expressed with spatial topological relationships as follows:

Topo1 = PL1 some trafficRule

• The class of Traffic behavior and Advanced Semantic behavior

The traffic behavior class corresponds to the activity behavior. The traffic behavior in port water areas can be divided into nine types of activity behavior, and the advanced semantic behavior can be divided into five types of process behavior and three types of event behavior, which can be seen in Figure 5.

Representation of behaviors with different semantic scales in the port water traffic areas can be done in different ways using ontology. Simple behaviors, such as active behaviors, can be expressed using property-constrained axioms based on atomic and topological behaviors.

Taking the behavior of Entering the Fairway as an example, the sufficient and necessary conditions for cross_into_lane should be as follows:

Trajectory T intersects with the line of fairway or the line between end points of the former, resulting in an intersection point located on an atomic trajectory AT which belongs to T. The beginning point of AT is located on the inside of the fairway and the endpoint is located on the outside of the fairway.. Likewise, the behavior of approach_pier can also be presented in a same way. The activity behavior of cross_into_lane and approach_pier can be represented as Figure 6.

5.3. Complicated Behavior Expressions Using SWRL

Complicated behaviors, such as process behaviors and event behaviors, are difficult to express directly with property-constrained axioms due to complex intrinsic behavioral logic. For this reason, we introduce Allen’s algebraic theory to model the temporal relationship of behaviors and express their complex behavioral intrinsic logical relations leveraging SWRL. Specifically, advanced semantic behaviors, such as event behaviors, consist of ordered specific activities and process behaviors, and as these behaviors occur, it can be triggered and inferred whether the advanced semantic behavior occurs or not. As shown in Figure 7, the event behavior of Anchor is explicitly temporal and logical, in which behaviors of the blue rectangular box and the gray arrow box make up the Anchor event.

For an explicit explanation, the process of the Anchor preparing process is selected to illustrate how to formalize the behavior, which is described below.

Anchor preparing process: The behavior from the instant the ship enters the anchorage until the start of anchoring, it consists of a series of activities in preparing to anchor. It is based on the existing knowledge to infer the advanced behavior, but the activity of preparing to anchor is not easily identified. Therefore, we do not use it to deduce the process. However, it is also worth noting that the trajectory corresponding to the behavior of the process contains a series of trajectories of the activity behavior, i.e., from the trajectory reflecting the cross_into_anchorage behavior to the trajectory reflecting the activity of the first anchor activity. Since these two behaviors can be obtained computationally from the AIS-based preprocessing module, it is possible to determine whether the process occurs by judging whether the two behaviors occur sequentially, which can be expressed in SWRL as:

Anchor preparing process = cognition2:ship (?s) ^ cognition2:trajectory (?t) ^ cognition2:hasTraj (?s, ?t) ^ cognition2:metaTraj (?stra) ^ cognition2:comprises (?t, ?stra) ^ cognition2:Point (?p1) ^ cognition2:Point (?p2) ^ cognition2:hasBeginPoint (?stra, ?p1) ^ cognition2:hasEndPoint (?stra, ?p2) ^ cognition2:LA5 (?stra, ?p) ^ cognition2:anchorage (?p) ^ cognition2:hasSpeed (?p2, ?x) ^ swrlb:lessThanOrEqual (?x, “0.5” ^^ xsd:float) → cognition2:hasBehavior (?s, cognition2:anchor_preparing)

Similarly, other behaviors occurring in port areas can be stated in the same way, as shown in

Table 1. Selected SWRL rules for reasoning about advanced behavior.

Number	SWRL Rules	Description
1	hasTraj(?s, ?t) ^ ship(?s) ^ reflects(?t, ?b) ^ behavior(?b) → hasBehavior(?s, ?b)	HasBehavior
2	ship(?s) ^ trajectory(?t) ^ metaTraj(?stra) ^ comprises(?t, ?stra) ^ hasTraj(?s, ?t) ^ approach_pier(?stra) → hasBehavior(?s, Approach_pier)	Behavior of approaching pier
3	trajectory(?t) ^ hasBeginPoint(?t, ?p1) ^ hasEndPoint(?t, ?p2) ^ Point(?p1) ^ Point(?p2) ^ Instant(?t1) ^ Instant(?t2) ^ hasInstant(?p1, ?t1) ^ hasInstant(?p2, ?t2) → occursEnd(?t, ?t2)	The occur begin time of trajectory
4	ship(?s) ^ trajectory(?t) ^ hasTraj(?s, ?t) ^ comprises(?t, ?straj) ^ metaTraj(?straj) ^ leave_pier(?straj) ^ LA4(?straj, ?p) ^ pier(?p) → leavePier(?s, ?p)	Behavior of leaving pier
5	ship(?s) ^ trajectory(?t) ^ hasTraj(?s, ?t) ^ berth_activity(?t) → hasBehavior(?s, berthing)	Berthing
6	ship(?s) ^ hasTraj(?s, ?t) ^ trajectory(?t) ^ cross_outof_anchorage(?subtra) ^ comprises(?t, ?subtra) ^ metaTraj(?subtra) → hasBehavior(?s, Cross_outof_anchorage)	Behavior of crossing out of anchorage
7	ship(?s) ^ trajectory(?t) ^ hasTraj(?s, ?t) ^ comprises(?t, ?straj) ^ metaTraj(?straj) ^ cross_outof_anchorage(?straj) ^ anchorage(?p) ^ LA4(?straj, ?p) → crossOutofAnchorage(?s, ?p)	Object properties for cross out of anchorage
8	ship(?s) ^ trajectory(?t) ^ hasTraj(?s, ?t) ^ metaTraj(?strai) ^ metaTraj(?straj) ^ comprises(?t, ?straj) ^ comprises(?t, ?strai) ^ LA4(?strai, ?p) ^ LA4(?straj, ?p) ^ fairway(?p) ^ cross_into_lane(?strai) ^ cross_outof_lane(?straj) → hasBehavior(?s, sailing_alongwith_fairway)	Behavior of sailing along with fairway
9	Instant(?i1) ^ Instant(?i2) ^ inXSDDateTimeStamp(?i1, ?it1) ^ inXSDDateTimeStamp(?i2, ?it2) ^ swrlb:lessThan(?it1, ?it2) → before(?i1, ?i2)	Time order
10	ship(?s) ^ trajectory(?t) ^ hasTraj(?s, ?t) ^ metaTraj(?stra) ^ comprises(?t, ?stra) ^ Point(?p1) ^ Point(?p2) ^ hasBeginPoint(?stra, ?p1) ^ hasEndPoint(?stra, ?p2) ^ LA5(?stra, ?p) ^ anchorage(?p) ^ hasSpeed(?p2, ?x) ^ swrlb:lessThanOrEqual(?x, 0.5) ^ deceleration(?a) ^ reflects(?stra, ?a) → hasBehavior(?s, anchor_preparing)	Behavior of preparing to anchor

.

6. Case Study

In order to investigate the feasibility of the cognitive model, we take the scenario as the experiment case where ship behavior, such as arrival and departure events, occurred in a port to show how complicated behaviors can be deduced in a cognitive way.

Firstly, based on the navigational experience of seafarers in port traffic areas, the most common ship behaviors occurring in order in these areas can be divided into three layers in which they occur, as shown in Figure 8. The overall behavior can be considered as an event of ship arrival-departure, in which the event of anchoring, entering fairway, berthing and unberthing, and departure are most commonly occur in an orderly manner. Likewise, the process layer and the activity layer can be extracted and depicted as follows.

6.1. Data Processing

The paper uses the AIS data and geographic data from Xiamen port for March and April 2016, including ship trajectory, fairways, anchorages, and piers. First, we pre-process ship trajectories, including data sorting and interpolation. Then, the dynamic AIS data are matched with the ship name, MMSI, and ship type in the static database to achieve the acquisition of ship attributes. Furthermore, the name, functional attributes, and location information of geographical objects can be obtained from www.chinaports.com (accessed on 18 September 2020).

Protégé is an ontology modeling tool [32] and is used here to construct an ontology model of ship behavior perception. We use version 5.5.0 of the software, version 2.2.0 of the Pellet reasoner and version 2.0.9 of SWRL. In addition, the model imports the time ontology abovementioned to support reasoning about complex behavioral temporal relationships.

6.2. Trajectory Segments and Semantic Annotation

In order to reduce the computational complexity and improve the ontological reasoning efficiency, 20,000 AIS data of ships are extracted for validation in this experiment. Firstly, the trajectories of different ships are sorted out in order to obtain the trajectories of each ship. Secondly, the trajectories are divided into “moving-stop” segments based on the recognition of stopping points to realize the labeling of moving trajectories. On this basis, we separate the moving trajectory from the stop trajectory to complete the annotation of atomic trajectory and the further recognition of atomic behavior. Finally, the start and end points of the trajectory are marked according to the stop, start, and end points of the atomic trajectory.

In order to calculate the spatial topological relationship between trajectories and the environment, the paper introduces a library for topology calculation based on Python programming language—Shapely. Firstly, various geographical objects and ship trajectories are converted into the format of spatial data. Then, the topological relationships of these converted objects are calculated to obtain the DE-9IM metrics of the relationships between trajectories and geographic objects. Finally, the computed results are mapped with the corresponding trajectories to prepare for the semantic annotation of ship behavior.

After the data level preparation is completed, the semantic information needs to be added to the ontology. In order to realize the combination of data and semantic information in the Python environment, the paper introduces Owlready2, a python-oriented ontology programming module that adds the already computed semantic information to the data layer and can load and save ontology files for modification and inference.

Figure 9 shows the overall process of behavioral cognitive computation, semantic reasoning and querying, which can support knowledge queries of behaviors with different semantic scales.

6.3. Semanticization of Ship Behavior

After importing the data related to ship behavior cognition into the ontology, including the ship, its trajectory segments, and the relationship between them, ship behavior can be clearly depicted. Figure 10 shows the importing results of ship RENLONG and the details of its trajectory segments, such as the place and time of occurrence.

Specifically, Figure 10a shows the movement and stationary trajectory segmentation identified for the ship RENLONG based on the “move-stop” trajectory segmentation method. Figure 10b shows the different atomic trajectory segments contained in the motion trajectory segmentation RENLONG_100 and the end point of this trajectory. Figure 10c shows the topological and temporal properties of RENLONG_ 10010135_begin_point as the starting point of RENLONG_10010135, in relation to the geographic region. Figure 10d represents the topological properties of the RENLONG_100_9851 atomic trajectory segment with respect to the geographic region around it, as well as its beginning and end points. Figure 10e shows the temporal properties of the time period in which the RENLONG_100 trajectory segment occurs, where the property of “has beginning” indicates that the beginning point of the trajectory segment occurred at the moment 1460149090. Figure 10f shows the XSD date timestamp property corresponding to the moment 1460149090.

As mentioned in the previous section, the atomic and topological behaviors can be stated based on the property-constrained axioms. The first-order atomic and topological behaviors are defined in terms of sufficient and necessary conditional constraint axioms. When the second-order behavior or topological features satisfy the definition of the class of the corresponding behavior, they will be automatically derived and classified to the corresponding first-order atomic behavior. As shown in Figure 11, the trajectory segment XINHAIXIU_49_3699 is classified as the class of cross_into_lane. On the basis of simple semantic behaviors, complicated behaviors, such as behaviors of process and event, can be further deduced based on the rules stated using SWRL, as described in Section 5.3.

For an intuitive comprehension of deduced behaviors shown in Figure 11, the trajectory marked with corresponding behavior is visualized in Figure 12 that shows ship semantic behaviors after trajectory segmentation, spatial topology calculation, and semantic annotation. Figure 12a, Figure 12c, and Figure 12d show the ship’s semantic behavior at anchorage, fairway, and berth, respectively, while Figure 12b is a zoomed-in view of the behavior in Figure 12a. The segmented ship trajectory segments can be clearly identified in these images, as well as the annotated advanced ship semantic behavior, such as the ship’s approach to the anchorage, the ship’s exit from the anchorage, the ship’s approach to the channel, etc.

7. Results

7.1. Semantic Query

Based on this ontology model, users can execute semantic queries on behavior cognition, such as ship trajectory, behavior, occurrence time, and occurrence place. In addition, the behavior of changing speed, changing course, stopping, and so on can be obtained based on the query. The SPARQL language of the query is shown below, and the results of the query can be seen in Figure 13.

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

PREFIX owl: <http://www.w3.org/2002/07/owl#>

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

PREFIX cog: <http://www.semanticweb.org/song/cognition2#>

SELECT ?ship ?behavior ?trajectory ?metatraj WHERE { ?ship cog:hasTraj ?trajectory.

optional {?trajectory cog:reflects ?behavior. ?trajectory cog:comprises ?metatraj.}}

7.2. Semantic Reasoning

Semantic reasoning is the process of acquiring implicit knowledge by driving the reasoning function of an ontology. Figure 14 shows an example of semantic reasoning about the behavior of a ship at anchorage. The attributes with yellow background of RENLONG are based on the inference results, including the implicit behaviors generated by RENLONG, its trajectory segments, and the place where and the time when these behaviors took place.

The result of the inference shows that RENLONG has the behavior of an anchor event. The start time is the beginning moment of the cross_into_anchorage behavior, and the end time is the end moment of the cross_out_of_anchorage behavior. Likewise, the behaviors occurring in the fairway and the pier of the ship can be reasoned out like the reasoning process of ship behavior in the anchorage.

As can be seen in Figure 13, the value of the object property of RENLONG has behavior is cross_into_anchorage, Anchor, and anchoring_event, but it cannot be deduced to the scale to which the behavior specifically belongs, such as activity and event. For this reason, the ontology sets different scales of behavior for object properties describing the scale of ship behavior, such as the properties of leave_pier and crossIntoAnchorage, which can provide a computational basis for reasoning about complicated behaviors.

7.3. Discussion

The behaviors of ships navigating around anchorages, fairways, and piers are selected for property-constrained axiom-based reasoning with SWRL for complicated behavioral reasoning, respectively. The results show that desired semantic behaviors can be recognized, leveraging the inference mechanism of behavior ontology, including from simple semantic behaviors, such as atomic behavior to large scale ship behavior, such as event in port waters. Key information of ship behavior cognition can be characterized, proving the advantage and effectiveness of the model in recognizing ship semantic behaviors, especially complicated temporal behaviors.

In addition, the behavior of a ship can be expressed by the object property ‘has behavior’, and the behavior with different scales can be characterized by setting the corresponding object property to achieve the multi-scale behavior of the ship. On this basis, the SWRL rule can be used to achieve the progressive reasoning of behavior between different scales, which is in line with the human habit of behavior cognition. However, such an approach is too cumbersome and all the rules need to be added manually by the people with expert knowledge, which consumes a lot of resources.

8. Conclusions

To enable autonomous objects in waterborne transport systems to have the capability of reasoning about and recognize historical complicated ship behavior semantically based on the historical AIS trajectory data, this paper proposes a framework for constructing semantic models of multi-scale ship behavior in cognitive space to achieve automatic extraction of semantic behavior of ships from the data layer to the semantic layer. On the basis of multi-scale characteristics of ship behavior reflected in ship trajectories, combined with the logical way humans perceive complicated behaviors, the cognition of ship behavior by an intelligent supervision system can be seen as an all-encompassing cognition involving the object, time, place, and behavior of the occurrence of ship behavior. Therefore, based on our previous work, this paper introduces a multi-scale behavioral semantic representation model to support the intelligent supervisory system’s cognition of ship behavior in a multi-dimensional and multi-scale space. Using the logical reasoning capabilities of the ontology and the temporal ontology’s modelling basis for time, ship behavior, including both simple and complex behaviors, can be accessed driven by the knowledge representation and logical reasoning capabilities of ontology. This suggests that it is feasible and reasonable to model the behavior of ships at multiple scales in a human cognitive manner.

However, there are some points that need further improvement. First, the model relies heavily on domain knowledge and needs to be constructed by domain experts, leading to inefficient application in practical scenarios. In addition, the paper does not consider the probability of ship behavior, especially in the continuous process, which limits the effectiveness of behavior implementation. In addition, there are various navigation scenarios where infrastructure exists that needs to be identified by the autonomous objects themselves or be considered as variables for human operator input for further analysis, which also needs to be addressed or clarified in the future. What needs to be done in the future is how to quickly extract and transform the textual information obtained from the website for various navigation scenarios, such as navigation modes, mooring information, etc., into knowledge that can be processed and understood by the autonomous system and expand it into a knowledge base with some scenario migration capability to make it highly reusable in different scenarios.

Future work can focus on the following points: firstly, online modelling, and identification of ship semantic behavior based on ship trajectory data; secondly, based on the semantic annotation results of historical ship trajectory data, combined with data mining algorithms, further mining of ship behavior at different semantic scales in port waters from the semantic layer to obtain implicit knowledge of high level ship behavior semantics. Finally, extending the individual semantic behavior model to interactive behaviors between two or more vessels can support the safety supervision of the waterborne transport system.