Challenges for the evaluation of automated driving systems using current ADAS and active safety test track protocols

A number of public safety stakeholders have advocated for the application of traditional consumer-focused testing protocols (e.g., NCAP programs) for the evaluation of safety for Automated Driving Systems (ADSs). Even though test protocols only exist for some ADAS and active safety technologies (i.e. SAE Level 0, Level 1, Level 2), proposals for expansion and adaptation of these tests for ADS have been brought forth within the industry community. To gain practical insight into the types of challenges and limitations arising from the application of these existing test protocols to ADSs, the Waymo Driver™, a SAE Level 4 ADS, was the subject of a testing campaign that leveraged several of the most difficult currently available ADAS and active safety test procedures. The main challenge discovered was that while these ADAS and active safety tests are aimed at evaluating the systems’ collision avoidance behavior, most of these tests were unable to be evaluated as designed due to the increased capabilities of the Waymo Driver that prevented the vehicle from even entering into a conflict to begin with. Difficulties encountered included creating the type of occlusions envisioned in some test protocols due to the location and performance of the Waymo Driver’s sensor suite and insufficient information in the test procedure regarding the roadway and map information. For example, in the occluded vulnerable road user (VRU) scenarios, the Waymo Driver could sense the test target prior to it starting to move and could proactively slow down, resulting in the desired collision avoidance interaction in the scenario not being tested. To make the test conditions representative of the intended collision avoidance interactions in the test procedure, either extra vehicles and/or different vehicle types were used as the occluding vehicles (e.g., large trucks). Similarly for the car-to-car test, a larger obstructing lead vehicle was used for the cut-out test so the Waymo Driver could not see over the lead vehicle. Also, without specifying additional details for the roadway that were not in the original test procedure, the Waymo Driver would proactively slow down due to the presence of parked cars or other roadway features on the test track, such as intersections. Beyond these required modifications to enable the interactions described in the test procedure, additional optional modifications were made to the test to increase the difficulty of the test. For example, in the NCAP cut-in test, the distance at which the vehicle was cutting in was reduced from 7.5m to 3m to try to elicit collision avoidance behavior. For all the test runs, including those run to specification and those with modifications, the Waymo Driver was able to avoid collisions which would have resulted in the highest rating for this evaluation. The conclusion is that existing ADAS and active safety test protocols cannot be applied as-is for an ADS such as the Waymo Driver. The highlighted challenges, ranging from the need to heighten the difficulty of the proposed scenarios to the under-specification of certain aspects of the test protocol, result in ambiguous requirements for both the test developers, the test facilities, and the test site administrators. This further indicates that Level 0-2 systems need to be separately considered from Level 4 ADS, such as the Waymo Driver. Furthermore, the results of this testing calls into question the feasibility and utility of adapting ADAS and active safety test for ADSs.