An integrated framework with explainable models based on visible and infrared images for safety enhancement in harbour areas

Recent advances in machine learning and artificial intelligence, driven by improved technologies and methodologies, have led to the widespread use of complex deep learning models. However, the increasing complexity and nonlinearity of these models have led to a lack of interpretability, especially in critical applications such as monitoring and surveillance. This study proposes the implementation of an integrated system consisting of visual and infrared cameras installed in a tourist port in Sicily to enable continuous monitoring of boats in transit. This monitoring system aims to increase safety in the port by providing operators with relevant information that can facilitate situational awareness. A YOLOv8-based object detection model was fine-tuned for the localization and classification of different boat classes in both the visible and infrared domains. In addition, the explainability of the model was investigated with the aim of identifying the most salient features in the detection and classification phase, thus creating heatmaps that are subsequently interpreted to highlight the most important features of the vessels used for classification. The developed infrastructure was evaluated in terms of processing time (28.47 fps), detection and classification rate (91.2% and 99.9% for the visible and infrared cameras, respectively) and interpretability of the classification results and compared with other solutions, demonstrating the effectiveness of the proposed solution.