ABOUT THE SITUATIONAL AWARENESS AND COLLABORATIVE PERCEPTION MACRO-TOOL

Set of Situational Awareness and Collaborative Perception tools to obtain a comprehensive understanding of the environment in CCAM systems

Situational Awareness and Collaborative Perception macro-tool has the purpose to obtain a comprehensive understanding of the environment in which each of the different assets or devices of a CCAM system are.

The Situational Awareness and Collaborative Perception framework seamlessly integrates cutting-edge connected and intelligent infrastructures such as Roadside Units (RSUs) and in-vehicle sensors to gather detailed insights into the CCAM ecosystem.

Tools categories

It comprises three primary categories of tools: vehicle-centred tools, enabling vehicles to perceive and interpret their surroundings accurately, RSU-centred tools, dedicated to leveraging advanced and intelligent hardware and software within Roadside Units (RSUs) to gather and analyse data pertinent to the CCAM system’s environment, and tools for data fusion and situational awareness, aimed at facilitating the aggregation and fusion of shared perception data.

CCAM ecosystem tools

01.

Advanced and intelligent hardware and software tools for situational awareness

02.

Methods for integration of perception systems into vehicles & RSUs​

03.

Methods for aggregating and fusing shared perception data​

04.

Methods for raising situational awareness with shared perception data​

05.

Construct security and safety metrics

AI enhanced tools for data aggregation, fusion and situational awareness

CCAM sector leverage Vehicle-to-Everything (V2X) communications to integrate and process sensor data shared by traffic participants and infrastructure.

This process enhances overall situational awareness and collaborative perception by aggregating and fusing the collected data within a centralized system. Utilizing AI-enhanced algorithms, these tools evaluate the current environment to determine risk levels, thereby enhancing accuracy and bolstering cybersecurity measures to mitigate potential cyber-attacks.

Data aggregation and fusion tool

The data aggregation and fusion tool comprises two distinct sub-tools: the C-ITS Gateway and the Data Association and Fusion module. The C-ITS Gateway facilitates the transmission and reception of Cooperative Intelligent Transport Systems (C-ITS) messages, handling the aggregation of Cooperative Awareness Messages (CAMs) and Collective Perception Messages (CPMs). Meanwhile, the Data Association and Fusion module is responsible for correlating CAMs and CPMs with the output of the vehicle’s onboard perception system.

The C-ITS Gateway operates by receiving CAMs and CPMs and converting them into ROS2 messages. These messages are then correlated with the objects detected by the local object detection system, resulting in a comprehensive list of associated objects. This process enhances situational awareness by integrating data from various sources and facilitating more informed decision-making in connected cooperative and autonomous mobility scenarios.

Situational assessment module

The Situational Assessment module integrates input from multiple sources, including the environment model, V2X network, and ego vehicle network. Leveraging various AI models, it processes these inputs to detect anomalous situations effectively. Specifically, distinct AI models have been developed to detect anomalies, considering probabilities across scenarios:

  • Global Navigation Satellite System (GNSS) Loss-Spoofing: Detects instances where there is a disruption or manipulation in GNSS signals.
  • RSU-Vehicle Mismatch: Identifies disparities between data received from Roadside Units (RSUs) and data directly obtained from the vehicle.
  • Recognition of unexpected alterations in acceleration, deceleration, and steering angle values during vehicle motion.

By employing these AI models, the situational assessment module enhances situational awareness by promptly identifying and addressing potential anomalies, thus contributing to safer and more reliable connected cooperative and autonomous mobility systems.

RSU-centric tools for situational awareness

The development of tools for RSUs to attain situational awareness is key for the deployment of CCAM. SELFY project developed Situational Awareness and Collaborative Perception tools empowering RSUs to comprehend and respond to their surrounding environment effectively.

SELFY developed three RSU-centred tools for situational awareness, the Traffic Monitoring Tool, processing video data captured by the RSU’s cameras and the Sensor Fusion & Anomaly Detection Tool, processing lidar and radars data and detecting potential anomalies.

The output of the Sensor Fusion & Anomaly Detection Tool, representing a cohesive understanding of the RSU’s surroundings, is then relayed to the Threat Evaluation Tool, where the data is structured as a list of objects, with some objects potentially associated with anomaly data.

Traffic monitoring tool

The Traffic Monitoring Tool processes video data to detect objects. Detected objects are then provided to the Sensor Fusion and Anomaly Detection Tool, which fuses data from the different RSU sensors.

Therefore, the Traffic Monitoring Tool contributes to enhance the understanding of the Cooperative, Connected, and Automated Mobility (CCAM) environment.

Sensor fusion and anomaly detection tool

The Sensor Fusion & Anomaly Detection tool (SFAD) orchestrates the integration of data from Range Sensors and Smart Cameras installed on Road-Side Unit (RSU) infrastructure, culminating in a unified depiction of the surrounding environment. By assessing the coherence among various sensor sources within the RSU, it enables the Threat Evaluation Tool to discern internal anomalies effectively. Moreover, it scrutinizes the alignment between the RSU’s consolidated perception and data from Cooperative Intelligent Transport Systems (C-ITS), detecting anomalies pertinent to SELFY use cases.

This tool comprises several C++ ROS 1 nodes organized into two main groups: range sensor perception pipelines and the fusion node. The Sensor Fusion and Anomaly Detection Tool significantly enhances the RSU’s situational awareness, bolstering safety and reliability in connected cooperative and autonomous mobility ecosystems.

Threat evaluation tool

The Threat Evaluation Tool specializes in identifying and analyzing potential threats within the CCAM environment by utilizing data from Road Side Unit (RSU) sensors and Vehicle-to-Everything (V2X) communications.

Initially, it processes data from the RSU’s own sensors to detect any suspicious patterns. Subsequently, it compares these findings with the descriptions of objects obtained from V2X messages, ensuring robust threat detection and validation.

Once a threat is identified, the tool alerts the Vehicle Security Operations Center (VSOC) of any misbehaviors, playing a critical role in maintaining the security and efficiency of CCAM systems. In addition, it also shares its perception of the environment with nearby V2X agents through Collective Perception Messages (CPM), broadcasted via the Roadside ITS Station.

Vehicle-centric tools for situational awareness

The vehicle-centered tools for Situational Awareness and Collaborative Perception are designed to enhance the vehicle’s understanding of its surroundings through sophisticated vision-based techniques.

These tools leverage AI-enhanced algorithms to identify, categorize, and accurately position objects within the environment relative to the ego vehicle. By harnessing vision-based techniques, these algorithms provide the vehicle with detailed insights into its surroundings, facilitating informed decision-making in real-time.

Furthermore, the detected object information is seamlessly integrated into a Local Dynamic Map, enhancing the vehicle’s situational awareness. This integration allows the vehicle to receive and incorporate detections obtained through Vehicle-to-Everything (V2X) communication, thereby further enriching its understanding of the dynamic environment.

Vehicle-centered situational awareness tool

The Vehicle-Centered Situational Awareness Tool employs a camera mounted onboard the vehicle in conjunction with a pretrained YOLOv5 neural network to optimize the detection of 3D objects in the vehicle’s vicinity.

Camera-based algorithms, integrated into ROS2, capture frames and utilize YOLOv5 to identify objects, spatially locating them relative to the vehicle. Simultaneously, the tool utilizes the vehicle’s GPS signal, also received through ROS2, to georeference these object detections from local coordinates to global coordinates. A complementary module converts OpenStreetMaps data for visualization. V2X detections are seamlessly integrated into the Local Dynamic Map for enhanced situational awareness.