Next Article in Journal
A Data-Driven Diagnosis Scheme Based on Deep Learning toward Fault Identification of the Hydraulic Piston Pump
Next Article in Special Issue
Exploring the Pirate Attack Process Risk along the Maritime Silk Road via Dynamic Bayesian Network Analysis
Previous Article in Journal
Research on Energy Management for Ship Hybrid Power System Based on Adaptive Equivalent Consumption Minimization Strategy
Previous Article in Special Issue
Maritime Anomaly Detection for Vessel Traffic Services: A Survey
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JMSE
Volume 11
Issue 7
10.3390/jmse11071272
jmse-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Brief Report

Improving Maritime Domain Awareness in Brazil through Computer Vision Technology

by
Matheus Emerick de Magalhães
1,2,*,
Carlos Eduardo Barbosa
1,2,
Kelli de Faria Cordeiro
2,
Daysianne Kessy Mendes Isidorio
1 and
Jano Moreira de Souza
1
1
Program Systems Engineering and Computer Science, Alberto Luiz Coimbra Institute for Graduate Studies and Research in Engineering (Coppe), Universidade Federal do Rio de Janeiro (UFRJ), Rio de Janeiro 21941-972, Brazil
2
Centro de Análises de Sistemas Navais (CASNAV), Brazilian Navy, Rio de Janeiro 20091-000, Brazil
*
Author to whom correspondence should be addressed.
J. Mar. Sci. Eng. 2023, 11(7), 1272; https://doi.org/10.3390/jmse11071272
Submission received: 6 May 2023 / Revised: 26 May 2023 / Accepted: 9 June 2023 / Published: 23 June 2023
(This article belongs to the Special Issue Maritime Security and Risk Assessments)
Browse Figures
Versions Notes

Abstract

:
This article discusses the Brazilian maritime authority’s efforts to monitor and control vessels in specific maritime areas using data from the naval traffic control system. Anomalies in vessel locations can signal security threats or illegal activities, such as drug trafficking and illegal fishing. A reliable Maritime Domain Awareness (MDA) is necessary to reduce such occurrences. This study proposes a data-driven framework, CV-MDA, which uses computer vision to enhance MDA. The approach integrates vessel records and camera images to create an annotated dataset for a Convolutional Neural Network (CNN) model. This solution supports detecting, classifying, and identifying small vessels without trackers or that have deliberately shut down their tracking systems in order to engage in illegal activities. Improving MDA could enhance maritime security, including identifying warships invading territorial waters and preventing illegal activities.

1. Introduction

The Maritime Domain Awareness (MDA) concept and solutions were reformulated after the 9/11 terrorist attacks [1]. The MDA concept can be described as understanding the elements, mainly vessels and other assets, in a given region.
The expansion of the MDA is necessary for maritime authorities to monitor and control the strategic planning of maritime traffic and to support naval actions. These actions are mainly related to safety, security, and economic activities such as oil extraction, fishing, and freight transport.
Several solutions can be related to MDA activities. In summary, the main activities of trajectory analysis used to increase the MDA are vessel density [2], waterway, maritime waypoint [3], maritime route extraction [4], anomalous vessel detection [5], vessel collision avoidance [6], and others. In almost all solutions, vessel data is the central resource. Furthermore, the challenges found in extracting data knowledge lead to the use of multiple data sources to reach the maritime domain. Handling the challenges encountered in acquiring data knowledge and using multiple data sources is crucial for obtaining better insights [7,8].
In recent years, the maritime authority has faced several MDA challenges related to the knowledge and intelligence needed for maritime security [9]. Increasing automatic, self-reported, and computational vision data leads to a data-driven solution that increases maritime knowledge and provides intelligent solutions. Furthermore, many countries have proposed specific MDA frameworks to improve the MDA, such as Greece [10], Canada [8], and the Philippines [11]. On the other hand, the literature presents exhaustive and unreliable MDA solutions using generic methods, such as movement extraction, data aggregation, data analysis, computer vision, and deep learning, through automatic or synthetic data [12].
In 2021, the United States and Brazil signed a cooperation agreement to best bid for addressing regional and global threats. The maritime domain threats can be hitched to various scenarios, such as drug trafficking, illegal fishing, piracy, oil spills, and illegal immigration [13,14,15]. Bad actors can also falsify AIS messages to cover illegal activities [16].
This research identifies several challenges related to the absence, veracity, and completeness of the navigation records in monitoring and controlling the activities of vessels in territorial waters. Detection and classification of ships in MDA are crucial, but it requires data integration to feed the Convolutional Neural Network (CNN). The absence of veracity and data completeness in monitoring and controlling the activities of vessels in territorial waters, and an interest in regional cooperation and interoperability presented in this new scenario, motivated this work [17,18]. Our research contributes to the advancement of the field and helps push the boundaries of knowledge and foster innovation.
The objectives of this work are related to the creation of datasets of vessel images using navigation records from various data sources. The aim is to increase data labeling integrity and accuracy while supporting the development of neural network models for vessel detection solutions. Additionally, we cover different approaches, methods, technologies, and definitions presented in the literature related to this work. Therefore, this work aims at creating a navigation dataset and vessel records through the integration of heterogeneous data sources, the formation of an automated structure for the annotation of identified objects through the integration with navigation and vessel records, and the creation of a data-driven model for the classification of vessels that are present in Brazilian inland waters using a solution based on computer vision.

2. Vessel Detection Issues

The vessel-detection problem refers to identifying and locating maritime vessels in large bodies of water using various remote sensing techniques. This problem has gained increasing attention in recent years due to the growing importance of maritime transportation and the need for effective surveillance and monitoring of sea traffic. Various methods have been developed to address this problem, including satellite imagery, radar systems, and acoustic sensors. However, vessel detection remains a complex task due to factors such as adverse weather conditions, vessel size and speed, clutter, and other sources of interference. Advances in computer vision are helping to improve the accuracy and efficiency of vessel detection systems, but this remains an ongoing area of research and development.
Figure 1 and Figure 2 present images captured from the Tactical Image Console with Augmented Reality (Console de Imagens Táticas com Realidade Aumentada—CITRA). CITRA [19] is a system developed by the Brazilian Navy that allows an operator to visualize real images captured by a video surveillance camera combined with synthetic elements from sensors in the maritime Command and Control systems, using Augmented Reality techniques. In this visualization, CITRA combines the real images from the connected cameras with metadata obtained from a Command and Control system, functioning as a video situational awareness module. However, CITRA has limitations related to inconsistent data between the live feed from the camera and the vessel’s last known position in the Command and Control system.
In Figure 1, we highlight the limitation of a vessel that appears in the camera and is missing in the command and control system. In contrast, in Figure 2, we highlight a record in the command and control system representing an absent vessel—probably due to the latency of the positioning update. Therefore, this work proposes key steps for creating a data-driven framework to support increasing MDA detection and classification of vessel types using a computer vision-based solution, detecting and reducing these inconsistencies.

3. Maritime Domain Awareness (MDA)

3.1. Data Integration in MDA

Information on the behavior of vessels is the main situational element maritime authorities use to increase domain awareness. Several data sources provide the vessels’ movement records, including records from Automatic Identification Systems and other sources, such as Long-Range Identification and Tracking (LRIT) [20]. In general, movement records are self-reported by vessels or automatically identified using satellites. Maritime data integration aims to consolidate vessels’ movement data and information [21]. The importance of integrating data sources to increase the MDA is represented in the works of Bannister and Neyland [7] and Battistello et al. [8].
Consolidating movement and vessel data involves multiple data-processing activities [22]. The main activities are data collection and movement records associated with the various vessel identifiers, organization, temporal alignment, and data smoothing [23].
The heterogeneous data from multiple sources can provide a more complete picture of vessel movements and a better understanding of their behavior. A more complete understanding of vessel trajectory can overcome errors in vessel movement records, such as data loss, self-information error, malfunctioning radar systems, and problems that can occur while processing the data from these maritime traffic systems [24].
Developing heterogeneous data is geared towards developing domain, situation, and impact analysis strategies. Integrating and combining different maritime data sources create an efficient system of responding to the various data sources [11]. Table 1 presents the data sources in the Brazilian Maritime Area.
We aim to use computer vision on the CITRA camera view, which extracts potential vessels in the field of view. Then, the detected vessels are combined with the suitable data source (listed in Table 1) to increase the MDA.

3.2. Computer Vision in MDA

Computer vision appears as a solution to support human visual tasks to interpret and understand the maritime domain using Deep Learning Neural Network (DNN) models. The DNN models accurately identify and classify objects [12,25]. Yet, the DNN solutions have been used in several MDA challenges, including detecting and classifying vessel types using synthetic or real-world images.
A specific type of DNN, the Convolutional Neural Network, has revolutionized the art of object detection and recognition, achieving faster and more accurate results [26].
CNN uses far fewer weights in our deep network, allowing for significantly faster training. One of the most accurate, fast, and precise CNN available is the state-of-the-art framework You Only Look Once (YOLO) [27]. The YOLO meets the requirements in some proposal works (Redmon et al. [28] and Redmon and Farhadi [29]) as real-time processing and is robust to changes in lighting in images and, in non-synthetic images, keeping a simple neural network, using a single GPU with a smaller mini-batch size to train a model.
The literature presents other object detection algorithm ancestors and alternatives of the YOLO framework, such as Histogram of Oriented Gradients (HOG) [30], Fast R-CNN [31], Faster R-CNN [32], Single Shot Detector (SSD) [33], and RetinaNet [34].
CNN can adopt an approach based on transfer learning [35]. It starts with a pre-trained model for generic feature detection, usually using a generic dataset such as the Microsoft Common Objects in Context (MS COCO) [36]. The model is specialized with a fine-tuning strategy using a data-driven perspective.

4. The MDA Problem in Brazil

In Brazil, the need to monitor vessels and assets in territorial water such as the Blue Amazon (Amazônia Azul), shown in Figure 3, has also motivated the development of novel and specific solutions intended to increase the MDA [37]. In Figure 3, the lighter blue region represents Brazil’s Exclusive Economic Zone of 200 nautical miles. In contrast, the darker blue represents Brazil’s proposal to the UN, which extends the zone to the Continental Shelf, up to 350 nautical miles [38].
Monitoring territorial waters is necessary for efficient maritime policymaking. Maritime policymaking encompasses military and economic actions related to national security [37]. The maritime authority expresses great concern regarding incidents related to foreign vessel intrusions, drug trafficking, protected biological areas, oil spills, search and rescue operations, and illegal fishing. These situations highlight the importance of implementing effective monitoring protocols within the maritime domain. Tracking vessel activities across various maritime zones is crucial to protect marine environments.
Bunholi et al. [39] explained Brazil’s biological importance in summarizing its data collection system for all of its fisheries. In Brazil, more information is needed for the expansion of policies and actions in order to inspect illegal fishing activities, where the current lack of data makes it difficult to control the maritime domain better [40].
From a Brazilian maritime perspective, the security environment in the South Atlantic focuses on threats such as piracy, drug trafficking, and other forms of transnational crime [41,42]. In the military domain, the disclosure of information regarding maritime security is subject to restrictions and limitations due to the classified nature of military information. However, the intrusion of foreign entities into territorial waters remains a constant concern for maritime authorities [43].

5. CV-MDA: A Computer Vision-Based Framework to Improve Maritime Domain Awareness

This section proposes three layers in an architectural framework for broad MDA, called CV-MDA, presented in detail in Figure 4. The Data Acquisition Layer conducts a unique vessel identification parameter and the vessel’s motion image data acquisition. The Integration Layer establishes the vessel movement dataset, and the Detection and Classification Layer proposes a fine-tuned transferred learning approach to a detection and classification solution. The following subsections describe the tools chosen for developing the framework.

5.1. Data Acquisition Layer

This work used traffic open data sources from Brazilian Maritime Traffic and tracked image data from Console Tactical Images. A heterogeneous data source can adopt specific vessel identification parameters in such sensor systems. We conducted a study with 25,618,290 navigation records from eight data sources in a 60-day horizon in order to understand and establish a unique vessel identification parameter.
The three main identifiers used for vessel identification are International Maritime Organization Number (IMO) [40], International Radio Call Sign (IRIN) [44], and Mari-time Mobile Service Identity (MMSI) [45]. In our data, several vessels had more than one identifier, as shown in Figure 5. However, 82.6% had all three identifiers. Furthermore, we associated them in order to obtain a unique vessel identification.
We collected images of the Brazil vessel registry and integrated them using the new vessel identification to provide some ground truth input to our object-annotated dataset. We also developed a specific tool to capture the vessel images from the maritime area, presented in Figure 6. The main objective was to capture and store real-world vessel images in a specific Brazilian area to build a dataset and train our CNN model.
We were able to use a data augmentation solution to improve our object-annotated dataset. Data augmentation is a strategy that enables slight variations to improve the diversity and volume of the image dataset [46].

5.2. Data Integration Layer Based on Computer-Vision

In this section, we propose correlating vessel registry, movement data, and real maritime images to label a training dataset to build a fine-tuned CNN approach model to detect and classify vessels and support a wide MDA. The proposed model—shown in Figure 7—provides a maritime data integration solution to build an automatically annotated Brazilian vessel dataset. Once the images were captured, we applied different computer vision approaches for enhancing the detection and pre-classification of vessels on the sea. We proposed using YOLO in the current implementation with an open-source CNN framework architecture called Darknet.
After submitting the images to the model, the pre-trained CNN returned an object’s detection boundary box properties (location) and a class (boat) confidence result in a found object, as presented in Figure 8.
On the other hand, we correlated the movement data sources (self-automatic and automatic) with vessel registries. We created stay points corresponding with the movement spatial coordinates and built a boundary box around them. Therefore, vessel type information was associated with delimited boxes.
Both boundary boxes, from image and movement data, were used to create an annotated dataset. We compared the bounding boxes in a specific time window using an Intersection over Union (IoU) approach to label the images with vessel type from a movement data source. The IoU model is shown in Figure 9.
An IOU threshold determined a correspondence between the boundary boxes to create an annotated image with vessel type. The flow of successful correspondent boundary boxes of a specific vessel was used to build a sequenced ground truth vessel labeled dataset in the Brazilian area. As a result, the accuracy of the combination could provide veracity and completeness to vessel label images. This data was used as an input source for a fine-tuned CNN approach to discovering and classifying vessel data in the next step.
This section may be divided by subheadings. It should provide a concise and precise description of the experimental results, their interpretation, and the experimental conclusions that we can draw.

5.3. Detection and Classification Layer

In this subsection, we propose transfer learning by using a fine-tuning strategy, shown in Figure 10. We chose this strategy to improve CNN generalization and specialization to build a fully connected layer to classify the object (vessel).
A generalization in transfer learning implies that we are taking the generic feature learned from a pre-trained network and applying it to another similar network. In the initial phase, we addressed this approach choice using image data consisting of patterns, common inputs, and generic feature detections. On the other hand, specialization referred to a fine-tuning phase to specialize the detection and obtain specific features.
In transfer learning, we used a pre-trained CNN to acquire low-level features. We could also use the CNN YOLOv5 with a trained vessel MS COCO dataset to obtain the basics proprieties (such as color, patterns, shapes, and edges), results of detection, and classification tasks. Next, in fine-tuning, we froze these low-level trains on the MS COCO and only retrained the high-level features with our labeled dataset. Finally, we replaced the classification layer with our vessel type classes.
We were able to measure the model’s performance using a mean Average Precision (mAP) metric. A threshold determined if the object detection and classification task was valid or not and was used to evaluate the performance of an object detection model. After all stages, we started the final classification step for enhancing the real-time captured images into a CNN.

6. Experiment

To showcase the outcomes of our framework, we conducted a straightforward experiment. Firstly, we employed vessel registry images and a pre-trained model to detect and outline the boat class within these images, as shown in Figure 8. Subsequently, we integrated the outlined images with the corresponding navigation records, incorporating the spatial coordinates and timestamps as parameters. This process is depicted in Figure 11, showcasing the integration.
Performing this integration, as demonstrated in Figure 12, we successfully assigned specific classes to the objects. Utilizing the model derived from annotated images using navigation records, we obtained images from a camera and subjected them to the vessel detection model. The labels “Ship1,…, ShipN” represent the identified vessel types. Upon examining the image, it becomes evident that the proposed model cannot detect certain vessels, while others may be misclassified.
Figure 13 presents an instance illustrating the process of automatic annotation used for labeling vessel classes. Some vessels could not be visually detected, resulting in their exclusion from the labeling process due to insufficient data sources. The colors assigned to the classes represent the various data sources employed to label each vessel. A challenge arises when attempting to detect vessels lacking sensor data, hindering the automatic annotation process.
We can address these limitations by incorporating a crowdsourcing layer into the CV-MDA framework. This additional layer aims to evaluate the detection results and assist in annotating the classes of objects that lack vessel records and, therefore, cannot be annotated based on navigation records, such as small vessels. By introducing crowdsourcing, it becomes possible to gather collective insights and annotations, improving the accuracy and completeness of the vessel classification process.
Using multiple integrated navigation data sources for automatically annotating vessel images in the Brazilian maritime region is a novelty and a significant contribution to the field. Combining navigation data from different sensors and navigation identifiers allows for more precise and efficient automatic annotation of vessel images. This automatic annotation surpasses the manual or semi-automatic approach commonly found in the literature, where human experts annotate. Automating the annotation process reduces the dependency on manual annotations, which can be time-consuming and prone to errors. Additionally, integrating multiple sources of navigation data provides a more comprehensive view of maritime activities in the region, enabling a more accurate understanding of the context and navigation patterns.
Our approach also considers the reality in the Brazilian maritime context, which is a relevant contribution. Each maritime region has its specific characteristics and challenges, and adapting the solution to the local needs and particularities is crucial for obtaining more effective results. Integrating sensor data, such as AIS, radar data, and other relevant sources, allows for obtaining more comprehensive and reliable information about vessels in the Brazilian maritime region. This data integration has many practical applications, such as maritime traffic monitoring, border control, maritime security, and strategic planning. The automatic annotation approach for vessel images, combining navigation data from multiple sources, represents a significant advancement compared to previous works focusing on manual or semi-automatic annotation. Our approach can provide higher efficiency, scalability, and accuracy in the detection and annotation of vessels.

7. Final Remarks

In conclusion, the Brazilian maritime authority relies on vessel monitoring and control activities to detect anomalies in the maritime domain, which could signify potential security risks or illegal activities such as drug trafficking and illegal fishing. This issue is particularly pressing in the Blue Amazon. In response, we have proposed a novel solution for Maritime Domain Awareness (MDA) based on computer vision. The maritime authority has deemed our technique efficient and valuable for detecting and classifying vessel types, thus supporting a comprehensive MDA.
This work has presented a solution to enhance Maritime Domain Awareness (MDA) by developing a framework for creating datasets of vessel images using navigation records from various data sources. Creating datasets is an important step in increasing the completeness and accuracy of data labeling. The proposed approach strengthens the capability to develop models applied in neural network solutions for vessel detection. Navigation records as the foundation for labeling make obtaining precise and comprehensive information about vessel types and movements possible. This precise information improves the efficiency of detection models, provides a solid foundation for decision-making, and enhances maritime domain awareness. By integrating diverse data sources, a more comprehensive and reliable understanding of the maritime scenario can be achieved, aiding in detecting anomalies, preventing illegal activities, and ensuring the security and surveillance of territorial waters. We have focused on computer vision applied to near-shore camera data. However, we can extend the framework with a few adaptations to real-time satellite images to achieve similar results in a larger area.
Moreover, we can adapt the proposed approach for monitoring other objects, supporting autonomous driving, detecting wildlife, and facilitating intelligent drone surveillance. However, creating annotated datasets can be challenging for small vessels such as fishing and sports vessels with limited movement records. In these cases, manual or semi-automatic labeling is still necessary to create annotated datasets.
In summary, incorporating multiple data sources alongside AIS offers several advantages over relying solely on AIS for ship-related information. These advantages include enhanced redundancy, as incorporating diverse data sources mitigates limitations and improves ship information’s reliability and availability. Additional data points from various sources provide supplementary information beyond AIS, enabling a broader range of ship-related data for more detailed analysis and decision-making. Moreover, different data sources capture information from unique sensors and technologies, offering diverse perspectives on maritime activities and facilitating better situational awareness. Specialized data streams cater to specific needs, such as environmental monitoring or security assessments, while cross-verifying and validating data from multiple sources improves the accuracy and reliability of data analysis. By leveraging these advantages, organizations can comprehensively and accurately understand maritime operations, improving safety, security, and efficiency.

Author Contributions

Conceptualization, M.E.d.M. and C.E.B.; methodology, M.E.d.M.; software, M.E.d.M.; formal analysis, M.E.d.M.; investigation, M.E.d.M.; resources, J.M.d.S.; data curation, D.K.M.I.; writing—original draft preparation, M.E.d.M.; writing—review and editing, M.E.d.M., C.E.B., K.d.F.C. and D.K.M.I.; visualization, M.E.d.M.; supervision, C.E.B., K.d.F.C. and J.M.d.S.; project administration, M.E.d.M. and J.M.d.S.; funding acquisition, M.E.d.M. and J.M.d.S. All authors have read and agreed to the published version of the manuscript.

Funding

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior—Brasil (CAPES)—Finance Code 001.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data are strictly used within the project but could be available upon request.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. National Commission on Terrorist Attacks. The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks Upon the United States (Authorized Audio Edition, Abridged); Norton, W.W., Ed.; National Commission on Terrorist Attacks Upon the United States: New York, NY, USA, 2011; ISBN 978-0-393-34215-4. [Google Scholar]
  2. Millefiori, L.M.; Zissis, D.; Cazzanti, L.; Arcieri, G. Scalable and Distributed Sea Port Operational Areas Estimation from AIS Data. In Proceedings of the 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW), Barcelona, Spain, 12–15 December 2016; pp. 374–381. [Google Scholar]
  3. Liu, L.; Shibasaki, R.; Zhang, Y.; Kosuge, N.; Zhang, M.; Hu, Y. Data-Driven Framework for Extracting Global Maritime Shipping Networks by Machine Learning. Ocean Eng. 2023, 269, 113494. [Google Scholar] [CrossRef]
  4. Jousselme, A.-L.; Iphar, C.; Pallotta, G. Uncertainty Handling for Maritime Route Deviation. In Guide to Maritime Informatics; Artikis, A., Zissis, D., Eds.; Springer International Publishing: Cham, Switzerland, 2021; pp. 263–297. ISBN 978-3-030-61852-0. [Google Scholar]
  5. Iphar, C.; Ray, C.; Napoli, A. Data Integrity Assessment for Maritime Anomaly Detection. Expert Syst. Appl. 2020, 147, 113219. [Google Scholar] [CrossRef]
  6. Rong, H.; Teixeira, A.P.; Guedes Soares, C. Ship Collision Avoidance Behavior Recognition and Analysis Based on AIS Data. Ocean Eng. 2022, 245, 110479. [Google Scholar] [CrossRef]
  7. Bannister, N.P.; Neyland, D.L. Maritime Domain Awareness with Commercially Accessible Electro-Optical Sensors in Space. Int. J. Remote Sens. 2015, 36, 211–243. [Google Scholar] [CrossRef]
  8. Battistello, G.; Gonzalez, J.; Ulmke, M.; Koch, W.; Mohrdieck, C. Multi-Sensor Maritime Monitoring for the Canadian Arctic: Case Studies. In Proceedings of the 2016 19th International Conference on Information Fusion (FUSION), Heidelberg, Germany, 5–8 July 2016; pp. 1881–1888. [Google Scholar]
  9. Zheng, D.X.; Sun, X.D. A Knowledge Acquisition Model in Maritime Domain Based on Ontology. In Proceedings of the Proceedings 2014 IEEE International Conference on Security, Pattern Analysis, and Cybernetics (SPAC), Wuhan, China, 18–19 October 2014; pp. 372–375. [Google Scholar]
  10. Giannakopoulos, T.; Vetsikas, I.A.; Koromila, I.; Karkaletsis, V.; Perantonis, S. AMINESS: A Platform for Environmentally Safe Shipping. In Proceedings of the 7th International Conference on PErvasive Technologies Related to Assistive Environments, Island of Rhodes, Greece, 27–30 May 2014; ACM: New York, NY, USA, 2014; pp. 45:1–45:8. [Google Scholar]
  11. Vicente, R.; Tabanggay, L.; Rayo, J.F.; Mina, K.; Retamar, A. Earth Observations for Goal 14: Improving Maritime Domain Awareness Using Synthetic Aperture Radar Imaging with Automatic Identification System in the Philippines. ISPRS—Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2020, XLIII-B3-2020, 215–219. [Google Scholar] [CrossRef]
  12. Breitinger, A.; Clua, E.; Fernandes, L.A. An Augmented Reality Periscope for Submarines with Extended Visual Classification. Sensors 2021, 21, 7624. [Google Scholar] [CrossRef]
  13. Bosilca, R.-L. The Use of Satellite Technologies for Maritime Surveillance: An Overview of EU Initiatives. Incas Bull. 2016, 8, 153. [Google Scholar]
  14. Lindley, J.; Percy, S.; Techera, E. Illegal Fishing and Australian Security. Aust. J. Int. Aff. 2019, 73, 82–99. [Google Scholar] [CrossRef]
  15. Sakhuja, V. Indian Ocean and the Safety of Sea Lines of Communication. Strateg. Anal. 2001, 25, 689–702. [Google Scholar]
  16. Iphar, C.; Napoli, A.; Ray, C. An Expert-Based Method for the Risk Assessment of Anomalous Maritime Transportation Data. Appl. Ocean Res. 2020, 104, 102337. [Google Scholar] [CrossRef]
  17. Soares, A.; Dividino, R.; Abreu, F.; Brousseau, M.; Isenor, A.W.; Webb, S.; Matwin, S. CRISIS: Integrating AIS and Ocean Data Streams Using Semantic Web Standards for Event Detection. In Proceedings of the 2019 International Conference on Military Communications and Information Systems (ICMCIS), Budva, Montenegro, 14–15 May 2019; pp. 1–7. [Google Scholar]
  18. Emmens, T.; Amrit, C.; Abdi, A.; Ghosh, M. The Promises and Perils of Automatic Identification System Data. Expert Syst. Appl. 2021, 178, 114975. [Google Scholar] [CrossRef]
  19. Centro de Análises de Sistemas Navais Console de Imagens Táticas com Realidade Aumentada (CITRA). Available online: https://web.archive.org/web/20230520030321/https://www.marinha.mil.br/casnav/?q=node/181 (accessed on 20 May 2023).
  20. Verma, B. Long Range Identification and Tracking (LRIT) Apropos Global Maritime Security. Marit. Aff. J. Natl. Marit. Found. India 2009, 5, 39–56. [Google Scholar] [CrossRef]
  21. El Faouzi, N.-E.; Klein, L.A. Data Fusion for ITS: Techniques and Research Needs. Transp. Res. Procedia 2016, 15, 495–512. [Google Scholar] [CrossRef] [Green Version]
  22. Steinberg, A.N.; Bowman, C.L. Revisions to the JDL Data Fusion Model. In Handbook of Multisensor Data Fusion; CRC Press: Boca Raton, FL, USA, 2017; pp. 65–88. [Google Scholar]
  23. García, J.; Guerrero, J.L.; Luis, A.; Molina, J.M. Robust Sensor Fusion in Real Maritime Surveillance Scenarios. In Proceedings of the 2010 13th International Conference on Information Fusion, Edinburgh, UK, 26–29 July 2010; pp. 1–8. [Google Scholar]
  24. Claramunt, C.; Ray, C.; Salmon, L.; Camossi, E.; Hadzagic, M.; Jousselme, A.-L.; Andrienko, G.; Andrienko, N.; Theodoridis, Y.; Vouros, G. Maritime Data Integration and Analysis: Recent Progress and Research Challenges. Adv. Database Technol.-EDBT 2017, 2017, 192–197. [Google Scholar]
  25. Kim, J.-H.; Kim, N.; Park, Y.W.; Won, C.S. Object Detection and Classification Based on YOLO-V5 with Improved Maritime Dataset. J. Mar. Sci. Eng. 2022, 10, 377. [Google Scholar] [CrossRef]
  26. Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
  27. Bochkovskiy, A.; Wang, C.-Y.; Liao, H.-Y.M. Yolov4: Optimal Speed and Accuracy of Object Detection. arXiv 2020, arXiv:2004.10934. [Google Scholar]
  28. Ribeiro, R.; Cruz, G.; Matos, J.; Bernardino, A. A Data Set for Airborne Maritime Surveillance Environments. IEEE Trans. Circuits Syst. Video Technol. 2019, 29, 2720–2732. [Google Scholar] [CrossRef]
  29. Redmon, J.; Farhadi, A. Yolov3: An Incremental Improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar]
  30. Surasak, T.; Takahiro, I.; Cheng, C.; Wang, C.; Sheng, P. Histogram of Oriented Gradients for Human Detection in Video. In Proceedings of the 2018 5th International Conference on Business and Industrial Research (ICBIR), Bangkok, Thailan, 17–18 May 2018; pp. 172–176. [Google Scholar]
  31. Girshick, R. Fast R-CNN. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7–13 December 2015; pp. 1440–1448. [Google Scholar]
  32. Jiang, H.; Learned-Miller, E. Face Detection with the Faster R-CNN. In Proceedings of the 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017), Washington, DC, USA, 30 May–3 June 2017; pp. 650–657. [Google Scholar]
  33. Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. SSD: Single Shot MultiBox Detector. In Proceedings of the Computer Vision—ECCV 2016, Amsterdam, The Netherlands, 11–14 October 2016; Leibe, B., Matas, J., Sebe, N., Welling, M., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 21–37. [Google Scholar]
  34. Wang, Y.; Wang, C.; Zhang, H.; Dong, Y.; Wei, S. Automatic Ship Detection Based on RetinaNet Using Multi-Resolution Gaofen-3 Imagery. Remote Sens. 2019, 11, 531. [Google Scholar] [CrossRef] [Green Version]
  35. Ahmed, I.; Ahmad, M.; Ahmad, A.; Jeon, G. Top View Multiple People Tracking by Detection Using Deep SORT and YOLOv3 with Transfer Learning: Within 5G Infrastructure. Int. J. Mach. Learn. Cybern. 2021, 12, 3053–3067. [Google Scholar] [CrossRef]
  36. Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft COCO: Common Objects in Context. In Proceedings of the Computer Vision—ECCV 2014, Zurich, Switzerland, 6–12 September 2014; Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T., Eds.; Springer International Publishing: Cham, Switzerland, 2014; pp. 740–755. [Google Scholar]
  37. Boraz, S.C. Maritime domain awareness: Myths and realities. Nav. War Coll. Rev. 2009, 62, 137–146. [Google Scholar]
  38. Thompson, N.; Muggah, R. The Blue Amazon: Brazil Asserts Its Influence across the Atlantic 2015. Available online: https://igarape.org.br/the-blue-amazon-brazil-asserts-its-influence-across-the-atlantic/ (accessed on 21 May 2023).
  39. Bunholi, I.V.; da Silva Ferrette, B.L.; De Biasi, J.B.; de Oliveira Magalhães, C.; Rotundo, M.M.; Oliveira, C.; Foresti, F.; Mendonça, F.F. The Fishing and Illegal Trade of the Angelshark: DNA Barcoding against Misleading Identifications. Fish. Res. 2018, 206, 193–197. [Google Scholar] [CrossRef] [Green Version]
  40. International Maritime Organization IMO Ship Identification Number Scheme. Available online: https://www.imo.org/en/OurWork/IIIS/Pages/IMO-Identification-Number-Schemes.aspx (accessed on 27 April 2022).
  41. Duarte, É.E.; Kenkel, K.M. Contesting Perspectives on South Atlantic Maritime Security Governance: Brazil and South Africa. South Afr. J. Int. Aff. 2019, 26, 395–412. [Google Scholar] [CrossRef]
  42. Medeiros, S.E.; Moreira, W.D.S. Maritime Co-Operation among South Atlantic Countries and Repercussions for the Regional Community of Security Practice. Contexto Int. 2017, 39, 281–304. [Google Scholar] [CrossRef] [Green Version]
  43. Phillips, C.E.; Ting, T.C.; Demurjian, S.A. Information Sharing and Security in Dynamic Coalitions. In Proceedings of the Seventh ACM Symposium on Access Control Models and Technologies, Monterey, CA, USA, 3–4 June 2002; Association for Computing Machinery: New York, NY, USA, 2002; pp. 87–96. [Google Scholar]
  44. International Telecommunication Union International Call Sign Series. Available online: https://www.itu.int:443/en/ITU-R/terrestrial/fmd/Pages/call_sign_series.aspx (accessed on 28 April 2022).
  45. U.S. Coast Guard Navigation Center Maritime Mobile Service Identity. Available online: https://www.navcen.uscg.gov/?pageName=mtmmsi# (accessed on 28 April 2022).
  46. Mikołajczyk, A.; Grochowski, M. Data Augmentation for Improving Deep Learning in Image Classification Problem. In Proceedings of the 2018 International Interdisciplinary PhD Workshop (IIPhDW), Swinoujscie, Poland, 9–12 May 2018; pp. 117–122. [Google Scholar]
Jmse 11 01272 g001 550
Figure 1. No Sensor Data to Vessel Identification.
Figure 1. No Sensor Data to Vessel Identification.
Jmse 11 01272 g001
Jmse 11 01272 g002 550
Figure 2. Low Sensor Data Veracity.
Figure 2. Low Sensor Data Veracity.
Jmse 11 01272 g002
Jmse 11 01272 g003 550
Figure 3. Blue Amazon—Adapted from Thompson and Muggah [38].
Figure 3. Blue Amazon—Adapted from Thompson and Muggah [38].
Jmse 11 01272 g003
Jmse 11 01272 g004 550
Figure 4. The CV-MDA Framework.
Figure 4. The CV-MDA Framework.
Jmse 11 01272 g004
Jmse 11 01272 g005 550
Figure 5. Vessel Identifications in Movement Data.
Figure 5. Vessel Identifications in Movement Data.
Jmse 11 01272 g005
Jmse 11 01272 g006 550
Figure 6. Tool for capturing vessel images.
Figure 6. Tool for capturing vessel images.
Jmse 11 01272 g006
Jmse 11 01272 g007 550
Figure 7. Using Computer-Vision on Maritime Data Integration Steps.
Figure 7. Using Computer-Vision on Maritime Data Integration Steps.
Jmse 11 01272 g007
Jmse 11 01272 g008 550
Figure 8. Detection Boundary and Pre-Classification.
Figure 8. Detection Boundary and Pre-Classification.
Jmse 11 01272 g008
Jmse 11 01272 g009 550
Figure 9. IoU Boundary Boxes.
Figure 9. IoU Boundary Boxes.
Jmse 11 01272 g009
Jmse 11 01272 g010 550
Figure 10. CNN Transfer Learning by Fine-tuning Approach.
Figure 10. CNN Transfer Learning by Fine-tuning Approach.
Jmse 11 01272 g010
Jmse 11 01272 g011 550
Figure 11. Vessel’s image outline process.
Figure 11. Vessel’s image outline process.
Jmse 11 01272 g011
Jmse 11 01272 g012 550
Figure 12. Object Annotated with Vessel Registry (left) and Object Pre-Trained Annotated (right).
Figure 12. Object Annotated with Vessel Registry (left) and Object Pre-Trained Annotated (right).
Jmse 11 01272 g012
Jmse 11 01272 g013 550
Figure 13. Object detection by the fine-tuned CNN.
Figure 13. Object detection by the fine-tuned CNN.
Jmse 11 01272 g013
Table
Table 1. Data sources in the Brazilian Maritime Area.
Table 1. Data sources in the Brazilian Maritime Area.
Data SourceDescriptionSensor TypePrivacy
AISAutomatic Identification System used for short-range coastal monitoring on shipsTerrestrialPublic
SISTRAMInformation Systems on Maritime Traffic in BrazilInformation SystemPrivate
SIMMAPMaritime Monitoring System to support oil-related activitiesTerrestrialPrivate
LRITLong-Range Global Monitoring SystemSatellitePublic
PREPSProgram for Tracking Fishing Vessels via SatelliteSatellitePrivate
VRMTC (TRMN)Virtual Regional Maritime TrafficTerrestrialPrivate
MSSISThe Maritime Safety and Security Information SystemTerrestrialPrivate
PRENAVProgram for Tracking Vessels operating in navigations regulated by the National Waterway Transport Agency (Agência Nacional de Transportes Aquaviários—ANTAQ)SatellitePrivate
CRTAMASRegional Traffic Center for the South Atlantic area (Brazil, Argentina, Uruguay, and Paraguay)TerrestrialPublic
MILITARESBrazilian Navy ShipsRadioPrivate
ORBCOMMOrbcomm’s second and third generation of satellite constellation, operating in Low Earth OrbitSatellitePrivate
Tietê-ParanáAIS Tietê-Paraná for navegable rivers.TerrestrialPublic
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Emerick de Magalhães, M.; Barbosa, C.E.; Cordeiro, K.d.F.; Isidorio, D.K.M.; Souza, J.M.d. Improving Maritime Domain Awareness in Brazil through Computer Vision Technology. J. Mar. Sci. Eng. 2023, 11, 1272. https://doi.org/10.3390/jmse11071272

AMA Style

Emerick de Magalhães M, Barbosa CE, Cordeiro KdF, Isidorio DKM, Souza JMd. Improving Maritime Domain Awareness in Brazil through Computer Vision Technology. Journal of Marine Science and Engineering. 2023; 11(7):1272. https://doi.org/10.3390/jmse11071272

Chicago/Turabian Style

Emerick de Magalhães, Matheus, Carlos Eduardo Barbosa, Kelli de Faria Cordeiro, Daysianne Kessy Mendes Isidorio, and Jano Moreira de Souza. 2023. "Improving Maritime Domain Awareness in Brazil through Computer Vision Technology" Journal of Marine Science and Engineering 11, no. 7: 1272. https://doi.org/10.3390/jmse11071272

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop