Writers: Mikko Tarkiainen, Pasi Pyykönen (VTT)

Automated driving technology is continually evolving, but there are still areas that require further development. One of the challenges is operating in harsh weather conditions. It is crucial to recognize situations where LiDAR performance is compromised, for example in rain, fog, or snowfall.

Moreover, automated functions can provide support during manual driving in specific circumstances. The integration of driver monitoring with vehicle environment perception and automated driving features can offer valuable assistance to the human driver when needed. This feature could be needed in heavy traffic situations, where the driver has covered long distances and relied on ADAS systems for extended periods while maintaining a constant speed.

These topics were explored in the NextPerception project and demonstrated by VTT, TTS and Modulight in the project final event in Eindhoven in May 2023. A video recording of combined demonstration is available (Youtube link). Let’s dive into the details presented in the video.

VRU detection and LiDAR performance estimation in harsh conditions

VTT’s vehicle, equipped with a 128-layer LiDAR sensor, approaches a pedestrian zebra crossing. The LiDAR system accurately identifies individual objects, including traffic signs and Vulnerable Road Users (VRUs) such as pedestrians. Traffic signs are distinguished with green markers, while VRUs are encapsulated within white bounding boxes in the point cloud. The system compares the locations of the clustered traffic signs with the predefined positions of traffic signs (or any other road infrastructure) stored in an HD-map.

In the scenario featuring light rain, both traffic signs are clearly visible to the LiDAR system, enabling effective detection of pedestrians on the zebra crossing using the LiDAR point cloud. The user interface (UI), for demonstration purposes, displays the status of the LiDAR performance as “clear”.

In the scenario with limited visibility, where weather conditions are unfavourable, the LiDAR system encounters challenges. Again, the location of clustered objects is compared to predefined locations of the reference traffic signs. In this scenario, not all traffic signs considered as reference landmarks are visible, or their visibility is significantly reduced for the LiDAR (poor weather conditions was “simulated” by removing a traffic sign). This information serves as an indicator of compromised LiDAR performance, leading to reduced VRU detection capabilities. Consequently, in such scenarios, automated driving may require a decrease in driving speed or a transition to manual control.

VRU detection with driver gaze recognition in Heavy Goods Vehicle (HGV)

TTS HGV incorporates two 32-layer LiDARs (the latest revision included three LiDARs to detect objects also behind the corners) for environment perception. Additionally, a Modulight LiDAR prototype is used to detect VRUs. The HGV is equipped with two driver monitoring cameras that continuously monitor the driver’s gaze direction.

During manual driving, the vehicle’s system processes the LiDAR point cloud, enabling the detection of all objects along the vehicle’s trajectory. In one specific instance, a VRU (pedestrian) on a zebra crossing is identified. In addition, the Modulight LiDAR prototype was demonstrated which detect pedestrians and cyclists in a situation where they were commuting together with other traffic. The Modulight LiDAR protype could successfully detect pedestrians and cyclists in front of, beside, and coming to the view from behind of a passenger vehicle.

When the HGV system detects a VRU in the vehicle’s trajectory, it compares the driver’s gaze direction with the VRU’s location. If the driver fails to notice the VRU, the system issues a warning sound alert. In the presented case, the driver’s inattention to the road and the possible collision with the VRU continues even after the warnings. To ensure safety, the system takes control of the vehicle, reducing speed and altering the driving trajectory towards a secure area, effectively avoiding a collision with the VRU. Ultimately, the vehicle system autonomously brings the HGV to a safe stop.